Overview

Matplotlib这个库对比起D3.js, Google chart之类的前端可视化工具来说, 美观度和交互性都有所欠缺, 但是对于学术类的图片, 前两个显的太浮夸. 这时候就需要适当调整Matplotlib, 使得绘图不那么古板.

字体

matplotlib中最蛋疼的东西大概就是字体了，主要的问题在中文的支持问题，还有字体格式的问题。

字体列表：

在默认情况下，中文字体会显示成框框:

1
2
3
4
5
6
7
8
9
10
11

fig, ax = plt.subplots()
for i, (k, v) in enumerate(nade_per_round):
    ax.bar(left=i, width= 0.8, height=v, color=([x / 255.0  for x in [245, 79, 41]]
                                                if k in winner else [x / 255.0  for x in [64, 89, 82]]))
plt.title("每局购买投掷物的数量", {'fontname':'Wawati SC','fontsize':18})
plt.ylim([9, 16])
ax.set_xticks([i + 0.5 for i, _ in enumerate(nade_per_round)])
ax.set_xticklabels([x[0] for x in nade_per_round])
ax.set_ylabel("nade count")
fig.autofmt_xdate()
fig.savefig("font-example-no-chinese.png", dpi=300)

每局购买投掷物的数量 直接变成了10个框框，这时最简单的方法就是直接指定字体了。

那么，有什么字体可以用？网络上可以找到一些简单的方案:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# 代码来源网络
from matplotlib.font_manager import fontManager 
import matplotlib.pyplot as plt 
import os 
import os.path 
fig = plt.figure(figsize=(12,8)) 
ax = fig.add_subplot(111) 
plt.subplots_adjust(0.04, 0.04, 0.99, 0.95, 0, 0) 
plt.xticks([]) 
plt.yticks([]) 
x, y = 0.05, 0.18 
fonts = [font.name for font in fontManager.ttflist if os.path.exists(font.fname) and os.stat(font.fname).st_size>1e6] 
font = set(fonts) 
dy = (1.0-y)/(len(fonts)/4 + (len(fonts)%4!=0)) 
for font in fonts: 
    t = ax.text(x, y, font+":"+u"你好", {'fontname':font, 'fontsize':14}, transform=ax.transAxes) 
    ax.text(x, y-dy/2, font, transform=ax.transAxes) 
    x += 0.25 
    if x >= 1.0: 
        y += dy 
        x = 0.05 
ax.set_xlabel(u'横坐标',{'fontname':'STKaiti'})
ax.set_ylabel(u'纵坐标',{'fontname':'STHupo','fontsize':20})
plt.show()

就可以看到这样一个字体列表：

嗯，虽然看上去不太和谐，但总算有个列表里，可是里面的东西为什么这么奇怪！
其实我只想要一个科学的中文字体和一个等宽字体!?
为什么列表里面没有PingFang SC?

Googling….

嗯，这里只是ttf格式的字体，不包括ttc或者otf的字体~ 那只好手动加载了。
macOS 上的字体文件夹的位置

比如说我要的PingFang SC就在/系统/资源库/Fonts/下面.

手动加载字体

目标：使用PingFang SC作为title的字体.

Googling …

很简单，font_manager 提供加载类

1
2
3
4
5

# 来源: http://stackoverflow.com/questions/7726852/how-to-use-a-random-otf-or-ttf-font-in-matplotlib
import matplotlib.font_manager as fm
f = "/System/Library/Fonts/PingFang.ttc"
prop = fm.FontProperties(fname=f)

然后在需要用到的时候, 指定一下，就可以了：

1

plt.title("每局购买投掷物数量", fontproperties=prop) #{'fontname':'simfang','fontsize':18})

对于大部分童鞋，应该都可以确定想要用的字体，或者在PS，Sketch上面试一下，找到合适的
这样就可以替换掉之前存在的框框了～

使用PingFang SC的title

推荐字体

建议任何非演示，科学绘图，请不要使用Matplotlib。Echarts, D3.js这些Web based工具会对绘图提供极大便利，接下来的交互设置也容易很多。另外，使用CSS指定字体也会比Matplotlib中方便很多.

一. 流程

1. 确定包的功能
2. 编写测试
3. 编写功能
4. 编写文档
5. Debug

测试

测试驱动开发(TDD), 在编写某一个功能之前先编写测试用例，然后迅速使测试工作，然后重构掉其中不合理的部分。对于一个小的工程，首先编写测试可以明确需求和目标，加快开发速度。

文档

在包内的每一个类和函数中，都应该加入docstring，详细描述该类/函数的具体参数，功能和返回值。在编写完代码之后，应该使用自动化的工具来生成文档。

Debug

Debug….Debu….Deb…De…D..D.

Question

Assume that we have 10000 genes, which is quantitied in normal ppl and turmor ppl. We have a algorithm to convert it to a bool table, than we want to pick up the combinations of gene have satisfied sensitivity and specificity

Sensitivity (also called the true positive rate, the recall, or probability of detection[1] in some fields) measures the proportion of positives that are correctly identified as such (e.g., the percentage of sick people who are correctly identified as having the condition).
Specificity (also called the true negative rate) measures the proportion of negatives that are correctly identified as such (e.g., the percentage of healthy people who are correctly identified as not having the condition).

First idea

Seem that intutive idea is like this

1. read bool table
2. generate combination?
3. check the Sen. and Spe.

use python’s itertools.combination to generate a iterator of combination of potential result and check it. However, even while handling 100 genes, the result may be tremendous. for example

1

5C100 = 75287520

The Language like Python or perl, handle these kind of loop was terriblly slow, one of my mate takes nearly 35 hours to finish a 5C100 recombination in Perl.

Is every combination needed?

After pick some genes, we actually have a bool matrix like this:

1
2
3

      |<-----normal---->|<----turmor----->|
g1    0   0   1   1   1   0   1   0   0   0
g2    1   1   1   0   1   1   0   1   1   1

than we can calculate the spe, filter the set whose spe or sen < 0.8:

1
2
3
4
5
6
7

spec, sen = 0, 0
spec = len([x for x in range(normal)
	if g1[x] and ge2[x] 
]) / normal
sen = len([x for x in range(normal, turmor)
	if g1[x] and ge2[x] 
]) / turmor

Here we feel diffcult to reduct the execute time of this part of codes. However, we find out that if one gene’s spe or sen < 0.8, than the combination won’t satisfied the requirements. So some improvements could be made.

Little improvements

so assume we have a list initial, than we can filter the unsatisfied single gene.

1

candidate = [gene for gene in initial if is_satisfied(gene)]

Than we run first combination, 2 C len(candidate), far less than the inital count.

Go deeper

when we get the result of 2 elements’s combination, we next want to generate the 3 element’s combination.

1
2
3
4
5
6

def combination(list1, list2):
	result = []
	for x in list1:
		for y in list2:
			list.append(frozenset(x + [y]))
	return set(result)

Cytoscape.js

Graph theory (a.k.a. network) library for analysis and visualisation

Official site

http://js.cytoscape.org/

Setup

like most drawing library, the setup processdure is same:

• include the libraries

  1	<script src="http://cdn.bootcss.com/cytoscape/2.7.8/cytoscape.js"></script>

• init with a canvas

  1	<div id="cy"></div>

• using a drawing config to create a cytoscape graphic.

  1	var cy = window.cy = cytoscape(config)

HelloWorld

using the setup example in the official site, combine with what we mentioned up, we can setup a hello world page:

code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>first_cytoscape</title>
    <script src="http://cdn.bootcss.com/cytoscape/2.7.8/cytoscape.js"></script>
</head>
<body>
    <div id="cy" style="width: 800px; height: 600px; background-color: #1c94c4"></div>
    <script>
        var config = {
            container: document.getElementById('cy'),
            elements: [
                { // node n1
                    group: 'nodes', // 'nodes' for a node, 'edges' for an edge
                    data: { 
                        id: 'n1', 
                        parent: 'nparent', 
                    },
                    scratch: {
                        foo: 'bar'
                    },
                    position: {
                        x: 100,
                        y: 100
                    },
                    selected: false,                    
                    selectable: true,
                    locked: false,
                    grabbable: true,                    
                    classes: 'foo bar' 
                },
                { // node n2
                    data: { id: 'n2' },
                    renderedPosition: { x: 200, y: 200 } 
                },
                { // node n3
                    data: { id: 'n3', parent: 'nparent' },
                    position: { x: 123, y: 234 }
                },
                { // node nparent
                    data: { id: 'nparent', position: { x: 200, y: 100 } }
                },
                { // edge e1
                    data: {
                        id: 'e1',
                        source: 'n1', 
                        target: 'n2'  
                    }
                }
            ],
            layout: {
                name: 'preset'
            },
            style: [
                {
                    selector: 'node',
                    style: {
                        'content': 'data(id)'
                    }
                },
                {
                    selector: ':parent',
                    style: {
                        'background-opacity': 0.6
                    }
                }
            ]
        };
        cy = cytoscape(config);
    </script>
</body>
</html>

result

next, will only provide the changed config.

config

the config, and showed in last section, describe what(the nodes and edges) and how(styles) in the map.

• elements, it contains the nodes and the edges

  1	elements: []

• layout, the layout methods, if you want to set the fixed position, using the preset:

  1 2 3

  layout: { name: preset, }

• style: a list of css-style tell cy how to render objects

  1

  style: []

config: elements

config: style

Sample:

宽度

使用next主题，在1080p屏幕下内容部分过窄。
按照: 修改主题style: Github Issue

??

本地jquery提示错误。

The IO performance of RPi 3B

The hardware of a RPi 3B is listed below, no surprise, just ordinary as his ancestor：

• 802.11n Wireless LAN
• 4 USB ports (usb 2.0)
• 40 GPIO pins
• Full HDMI port
• Ethernet port(100Mbps)
• Camera interface (CSI)
• Display interface (DSI)
• Micro SD card slot (now push-pull rather than push-push)

In this generation of RPi, this limitation of performance is mainly in its IO performance, The 100Mbps Ethernet, usb 2.0 and 802.11n Wifi. This kind of performance has kill is possibility to become a NAS, which I think should have at least 2 STAT, use3.0, 1Gbps Ethernet, and a powerful Soc to satisfy this kind of IO requirement. But to be a wireless router, RPi its enough.

Sniff the packet

At the beginning, I use scapy to capture the traffic through the RPi like:

1

sniff(iface='ppp0', prn=lambda x: self.handle(x))

however, its cpu usage is ultra hight to >100%, result in a high ratio of package loss. (~10% captured).
After that I find use layer 2’s socket and tcpdump may be better solution.

1

sudo tcpdump -i ppp0 -w out traffic.pcap

which i can save the traffic to a pcap file and handle it in scapy.

Handle the packet and get the address:

In this capture, we will get the ip address of the traffic, check if it is the local ip or board cast ip address. than, query the public api to get the ipaddress.

1
2
3
4
5
6

pkts = rdpkt("traffic.pcap")
results = []
for x in pkts:
	if IP in x:
		if not x[IP].dst[:7] == '192.168' and not x[IP][:10] == '255.255.255'
			results.append(x[IP].dst)

Overviews

• Inject a dylib to csgo_osx64
• Get the interface and method’s address
• Hook method
• Build up the project

PartI, Injection and Read module handle and method address

  To build a internal hack, we simply inject a Dynamic Link Library to the gaming process. Start a new Thread, and gather the information we care, and do something.
  In this part of the article, we start with the topic of Injection, which is a very normal task in windows and linux, luckily, there are few wheel we can use in macOS:

• OSX injection tutorial: Hello World
• A Example to show the Inject process
• And its repo provide the hook method

First step: Build the project

  Let us start with the project the second repo mention last section, and copy the third’s repo’s testdylib’s code to replace the second one. so now we get the project. however as valve currently update cogs to 64bit, we need to set 4 project to x86_64 in project setting. after this, our project look like this: there should be a image

1
2
3
4
5

// the code in install will run after the dylib's injection
void install()
{
	printf("loaded\n!");
}

Get the module address and Find CreateInterface:

  In windows we have GetModuleHandle(may be wrong..)to get the handle of a dll. in macOS, we have the dlopen from dlfcn.h and apple provide detailed docs:

• Dynamic Library Usage Guidelines
• OS X ABI Dynamic Loader Reference
  The methods we use are detail listed in the docs, what we want to do is simply call them to get the address

1
2
3
4
5

auto address_of_engine = dlopen("client.dylib", RTLD_NOW);
if (!address_of_engine){
    printf("Unable to read the address of client.dylib: %s\n", dlerror());
    exit(0)
}

  if the dylib has been added into the process’s scope, what is the situation what we meets here, the method will return the handle(address of the dylib in the memory).
  After we get the address of the module, we start to ask where is CreateInterface which is the function to get the interface(like entity_list, engine_trace) which module expose to the extern.
use the dlsym to the the address:

1
2
3
4
5
6
7
8

// define the CreateInterface first
typedef void* (*CreateInterface)(const char*, int *);
// next call method dlsym
auto clintFactory = (CreateInterface)dlsym(address_of_engine, "CreateInterface");
if (!address_of_create_interface){
    printf("Unable to read the address of CreateInterface: %s\n", dlerror());
}

Use *Factory to get the Interface:

  After we get the ClientFactory and EngineFactory we have to get the useful object we need when making a hack. like this

auto entity_list = (IClientEntityList *)clientFactory(“VClientEntityList003", nil);

After doing this, we get the basic to making a Internal Hack.

Hello World

首先来尝试一下官方的例子:

1
2
3
4
5
6
7
8

# 在IPython环境下
from echarts import Echart, Legend, Bar, Axis
chart = Echart('GDP', 'This is a fake chart')
chart.use(Bar('China', [2, 3, 4, 5]))
chart.use(Legend(['GDP']))
chart.use(Axis('category', 'bottom', data=['Nov', 'Dec', 'Jan', 'Feb']))
chart.plot()

兼容IPython

常用图表: 饼图，柱状图，折线图和散点图

饼图:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from echarts import Echart, Legend, Bar, Axis, Pie
chart = Echart('Map Picks in MLG', 'In group A')
chart.use(Pie('Map Picks in MLG', 
              [{'value': 2, 'name': 'de_dust2'},
               {'value': 3, 'name': 'de_cache'},
               {'value': 12, 'name': 'de_mirage'},
               {'value': 9, 'name': 'de_inferno'}
              ],
            radius=["50%", "70%"])
         )
chart.use(Legend(["de_dust2", "de_cache", "de_mirage", "de_inferno"]))
del chart.json["xAxis"]
del chart.json["yAxis"]
chart.plot()

结果:

柱状图:

1
2
3
4
5
6
7
8

from echarts import Echart, Legend, Bar, Axis
chart = Echart("The Winrate of Tyloo and VG in this year", "The data is faked")
chart.use(Bar("Tyloo", ["50", "72", "64", "42", "30", "72"], ))
chart.use(Bar("VG", ["55", "48", "56", "63", "70", "55"]))
chart.use(Axis("category", "bottom", "map", ["de_dust2", "de_cache", "de_mirage", "de_cbble", "de_nuke", "de_cbble"]))
chart.use(Legend(["Tyloo", "VG"], position=('center', "bottom")))
chart.json
chart.plot()

散点图:

1
2
3
4
5
6
7
8

import numpy
from echarts import Echart, Legend, Scatter, Axis
data = numpy.random.uniform(0, 100, 100).reshape(50, 2).tolist()
chart = Echart("Random Number Distribute", "just a rundom number")
for x in data:
    chart.use(Scatter("{}".format(numpy.random.uniform(0, 100, 1).tolist()[0]), [x]))
chart.use(Legend(["data"]))
chart.plot()

结果:

折线图

1
2
3
4
5
6
7
8
9
10
11
12
13

from echarts import Echart, Legend, Line, Axis, Tooltip
import numpy as np
import math
data = [[round(t, 4) for t in np.random.uniform(15 * x, 25 * (x + 1), 7).tolist()] for x in range(4)]
chart = Echart("The expression level in four organsim", "random data")
organism = ["Spleen", "Skin", "HK", "Jill"]
for x in range(4):
    chart.use(Line(organism[x], data[x]))
chart.use(Axis('category', 'bottom', data=["0H", "6H", "12H", "2Day", "3Day", "5Day", "7Day"], boundaryGap=False))
chart.use(Axis('value', 'left'))
chart.use(Legend(data=["Spleen", "Skin", "HK", "Jill"], position=('center', 'bottom')))
chart.use(Tooltip())
chart.plot()

结果:

问题:

• 嵌入IFrame大小问题width:, height: ?。
• axis 无法隐藏问题，特别是在饼图中。
• 没有代码提示。
• 交互设置困难。

3D模型插件

长久以来，生物学的三维模型通常都是通过Java插件来显示，这个比Adobe Flash还要古老的Applet简直是古董级的东西，直到近期才开始使用原生技术。

3Dmol.js

这是一个基于WebGL的Javascript库，支持pdb, sdf, mol2, xyz, 和cube这些完全不认识的格式。没错，生物界可真的是封闭的很，每个人都喜欢提出一个新的格式，虽然很多都是基于XML的存在。

Nicholas Rego and David Koes 3Dmol.js: molecular visualization with WebGL Bioinformatics (2015) 31 (8): 1322-1324 doi:10.1093/bioinformatics/btu829

HOWTO

和大部分javascript库一样，只需要一个canvas，然后就可以在上面显示一些模型了，使用非常简单:

1

<div id="container-model" class="mol-container"></div>

然后初始化:

1
2
3
4

var element = $("#container-model")
var config = { defaultcolors: $3Dmol.rasmolElementColors, 
		backgroundColor: 'white' };
$3Dmol.createViewer( element, config );

最后加载一个模型：

1
2
3
4

v.addModel( data, pdb_link );
v.setStyle({}, {cartoon: {color: 'spectrum'}});
v.zoomTo();v.render();                                                 
v.zoom(1.2, 1000);

然后就可以了。

结果:

Interactive:

Wireless Freedom

Openwrt 是一个运行在MIPS, ARM, X86等各类嵌入式设备上的Linux系统。它的文件系统是可写的，意味着修改系统不需要重新编译整个系统。
开发者可以交叉编译软件到Openwrt，也可以使用它自带的opkg包管理系统安装软件/库。

1
2

opkg update
opkg install libpcap

没错，这就是一个liunx!

只是性能有所不足而已： 典型CPU

• MT7620: 580MHZ, MIPS, Single-Core

• AR71xx: MIPS, Single-Core, <800MHZ

典型RAM/FLASH

• FLASH: 4M, 8M, 16M

• RAM: 16M(天坑), 32M(主流), 64M(主流), 128M, 256M

硬件支持列表:

Openwork Wiki, 好像除了tplink新出的坑爹货，其他基本都可以刷openwork的系统。

路由器

路由器无论搭载什么系统, 首要要求就是稳定性, 稳定性好的, 才能讨论五花八门的高级功能。Openwrt作为一个老牌的开源路由器系统，它的稳定性比起那些MT762x之流的智能路由器(极路由1s, 360智能路由, 小米路由mini, 联想newwifi mini)高的不是一条街。和那些TP-LINK早期用料足的路由器在一个水平段(如WR841N v5)。
如何选择一个稳定的路由器:

• TP-LINK远古时期的路由器, 确定能刷Openwrt, 缺点是很多都是二手。

• 一些智能路由器，用的都是类似的方案(ODM), soc性能也过得去，刷了Openwrt也没用什么问题。

  基本用途

  整理一下思路，从连上网之前到连上网之后，到高级用途。

  联网前: 认证

  高校大量的openwrt路由器，基本都是为了绕过那些校园网的认证而刷的，神州数码, DrCOM, iNode之流，不仅客户短写的好像上世纪80年代的风格，尤其是图标，XP下都嫌难看，而且还限制wifi分享软件。所以刷openwrt跑认证就成了一种刚需。
  开源的客户端:

• DrCOM: DrCOM-generic

• 神州数码: zdclient

联网后

1. 翻墙: 前有PPTP，OpenVPN被GFW处理的服服帖帖，现在主流出墙基本都是shadowsocks。在路由器上配置ss, 并且使用PAC判断网站是否在墙外，并且自动代理，是目前最好的出墙方案。
2. 多播: 如果测试本地运营商没有限制宽带多播的话，可以尝试虚拟WAN口多播叠加带宽。(移动好像一直都不可以)
3. NAS, 离线下载: Openwork极少有带STAT或者USB3.0接口，部分带USB2.0接口的路由器IO性能不佳，加上孱弱的SOC性能，基本上只能处理百兆网口的下载流量，由于USB口性能的限制，很多时候还达不到10MB/s(参考Raspberry 1代的usb和有限网口)。所以做NAS就一鸡肋，聊胜于无。但是离线下载一些720P的视频还是不错的。 配置transmission
4. 去广告:…..

Python

如果要在Openwrt上写C程序，还是比较麻烦的。还要找一个linux的机器交叉编译，光配置和下载依赖包，可能就要一个晚上的时间，加上编译系统的3 4个小时，是一件费时费力的过程。况且C程序编程还十分繁琐。所以能运行Python就好了。

1
2
3
4

# 安装
opkg update
opkg install python-simple
opkg install python-codec

大致会消耗2.7MB的空间，对于8M Flash的路由器来说应该没有太多的压力。之后就可以使用socket模块愉快的抓包了。

Scut python版Drcom认证：

• DrCOM For Scut

• Scapy版

Scapy

毕竟在包分析方面，用Raw socket写的程序还是比不上成熟的网络分析工具Scapy，所以安装一个scapy对包分析还是很有帮助的。安装过程：

1
2
3
4
5
6
7
8
9

opkg update
opkg upgrade tar wget
opkg install python tcpdump unzip
wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz
tar -xvf scapy-latest.tar.gz
cd scapy*
python setup.py install
cd ..
rm -rf scapy*

抓一波包压压惊。

1

sniff(iface='ppp0', prn=lambda x: x.summary())

HttP Server

不知道能不能跑得动…