前言
这是个在写计算机网络课设的时候碰到的问题，卡了我一天，所以总结一下。
其实在之前就有用requests写过python爬虫，但是计算机网络要求更底层的实现，刚好我看到了[这篇文章]1结果发现他就是用socket来实现的请求，所以就学习了。
本来也觉得应该不难，毕竟就是建立tcp连接。
原网站的例子如下：
def fetch(url): sock = socket.socket() # 建立socket sock.connect(('xkcd.com', 80)) # 远程连接 request = 'GET {
}
HTTP/1.0rnHost: xkcd.comrnrn'.format(url) # 构建请求 sock.send(request.encode('ascii')) # 向socket发送数据 response = b'' chunk = sock.recv(4096) # 从socket接收数据 while chunk: response += chunk chunk = sock.recv(4096) # Page is now downloaded. links = parse_links(response) q.add(links)
我选择爬取的网站是链家，当然也看了很多其他例子,还用fiddler抓包，把headers整个放了上去，首先是参考了这个文章：https://segmentfault.com/a/1190000005126160 ，该文章介绍到：
Python通过socket发送http请求
我们以访问百度主页为案例，使用socket发送http请求。
import sockets=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect(('www.baidu.com',80))s.send('''GET https://www.baidu.com/ HTTP/1.1Host: www.baidu.comConnection: keep-aliveAccept: text/html,application/xhtml+xml,application/xml;
q=0.9,image/webp,*/*;
q=0.8Upgrade-Insecure-Requests: 1User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36Accept-Language: zh-CN,zh;
q=0.8''')buf=s.recv(1024)while len(buf):print bufbuf = s.recv(1024)
基于socket进行http编程在于请求参数的可控性更好，但是难度相应就更大了。以上发送的数据是直接从fiddler抓包器里面复制的。
基于以上，写出如下代码：
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect(('www.baidu.com',80))s.send('''GET / HTTP/1.1Host: zh.lianjia.comConnection: keep-aliveCache-Control: max-age=0Upgrade-Insecure-Requests: 1User-Agent: Mozilla/5.0 (Windows NT 10.0;
Win64;
x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36Accept: text/html,application/xhtml+xml,application/xml;
q=0.9,image/webp,image/apng,*/*;
q=0.8Referer: https://www.baidu.com/link?url=4J5Kx--GLdLFESJhkfRePU8Ac_0agnTcOtB-b3kfnX8VNdZ_6TPqOyJGKVXkTczg&ck=6140.3.83.296.315.287.208.155&shh=www.baidu.com&sht=94886267_hao_pg&wd=&eqid=af98b98700060b77000000065aef0524Accept-Encoding: gzip, deflate, brAccept-Language: zh-CN,zh;
q=0.9,en-CA;
q=0.8,en;
q=0.7Cookie: lianjia_uuid=ce61c41c-25b0-46d6-a0a0-d57a75ee8706;
UM_distinctid=1631f588055f9-0286722badd3ec-b34356b-1fa400-1631f58805657f;
_ga=GA1.2.43397143.1525239286;
_smt_uid=5ae94e02.558be516;
_jzqx=1.1525248800.1525335927.1.jzqsr=zh%2Elianjia%2Ecom|jzqct=/ershoufang/xiangzhouqu/.-;
_jzqc=1;
_jzqckmp=1;
_gid=GA1.2.1028411676.1525594529;
select_city=440400;
all-lj=c60bf575348a3bc08fb27ee73be8c666;
_qzjc=1;
CNZZDATA1254525948=963210960-1525238218-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608956;
CNZZDATA1255633284=1054798284-1525238580-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608969;
lianjia_ssid=c046ddb3-3e66-4809-998a-52ade335fdfc;
_qzja=1.1070225156.1525239298260.1525603274282.1525613866775.1525609113492.1525613866775.0.0.0.92.9;
_qzjto=29.3.0;
_jzqa=1.3750161754444366000.1525239284.1525603274.1525613867.9;
_jzqy=1.1525239284.1525613867.3.jzqsr=baidu.jzqsr=baidu;
Hm_lvt_9152f8221cb6243a53c83b956842be8a=1525607433,1525607626,1525609113,1525613867;
Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1525613867;
_qzjb=1.1525613866775.1.0.0.0;
_jzqb=1.1.10.1525613867.1;
CNZZDATA1255604082=964175865-1525237915-https%253A%252F%252Fwww.lianjia.com%252F%7C1525612833''')
结果总是报400(Bad Request) ，这个地方卡了很久，最后解决办法是一条一条发送，每一条后面加rn。
sock = socket.socket() sock.connect(('zh.lianjia.com', 80)) sock.send('GET /ershoufang/ HTTP/1.1rn'.encode()) sock.send('Host: zh.lianjia.comrn'.encode()) sock.send('Connection: keep-alivern'.encode()) sock.send('Cache-Control: no-cachern'.encode()) sock.send('Accept: text/html,application/xhtml+xml,application/xml;
q=0.9,image/webp,image/apng,*/*;
q=0.8rn'.encode()) sock.send('Upgrade-Insecure-Requests: 1rn'.encode()) sock.send('User-Agent: Mozilla/5.0 (Windows NT 10.0;
Win64;
x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36rn'.encode()) sock.send('Accept-Encoding: gzip, deflate, brrn'.encode()) sock.send('Cookie: lianjia_uuid=ce61c41c-25b0-46d6-a0a0-d57a75ee8706;
UM_distinctid=1631f588055f9-0286722badd3ec-b34356b-1fa400-1631f58805657f;
_ga=GA1.2.43397143.1525239286;
_smt_uid=5ae94e02.558be516;
_jzqx=1.1525248800.1525335927.1.jzqsr=zh%2Elianjia%2Ecom|jzqct=/ershoufang/xiangzhouqu/.-;
_jzqc=1;
_jzqy=1.1525239284.1525594526.2.jzqsr=baidu.jzqsr=baidu|jzqct=%E9%93%BE%E5%AE%B6;
_jzqckmp=1;
_gid=GA1.2.1028411676.1525594529;
Hm_lvt_9152f8221cb6243a53c83b956842be8a=1525594526,1525594536,1525594804,1525595210;
select_city=440400;
all-lj=c60bf575348a3bc08fb27ee73be8c666;
_qzjc=1;
lianjia_ssid=99306d63-8ee5-a53c-a740-2d3021f3db2f;
CNZZDATA1255604082=964175865-1525237915-https%253A%252F%252Fwww.lianjia.com%252F%7C1525602095;
_jzqa=1.3750161754444366000.1525239284.1525594526.1525603274.8;
CNZZDATA1254525948=963210960-1525238218-https%253A%252F%252Fwww.lianjia.com%252F%7C1525603556;
CNZZDATA1255633284=1054798284-1525238580-https%253A%252F%252Fwww.lianjia.com%252F%7C1525603557;
Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1525606057;
_jzqb=1.9.10.1525603274.1;
_qzja=1.1070225156.1525239298260.1525597069547.1525603274282.1525605398368.1525606071025.0.0.0.86.8;
_qzjb=1.1525603274282.9.0.0.0;
_qzjto=23.2.0rnrn'.encode())
结果总是重定向，状态码301！找了好久都不知道什么原因，而且直接在浏览器网址栏输入网址，用fiddler抓包也没有抓到状态为301的包。最后使用fiddler的composer输入http://zh.lianjia.com/ershoufang 就抓到了301和200，其中200的地址就是https://zh.lianjia.com/ershoufang，如下图所示。
这下知道原因了，是http和https的区别。(其实301状态码的时候应答部分的Location可以观察到，但是一个s太不显眼了所以我没有注意到，导致卡了很久)
接下来只要知道如何发送https请求就好了。下面是代码，主要是更改建立socket和连接的部分。注意端口号为443。参考文章在这里
sock = ssl.wrap_socket(socket.socket()) sock.connect(('zh.lianjia.com', 443))
感觉自己很多地方了解的不够深入，暂时学校也没有讲到应用层。到时候再研究研究，如有错漏欢迎指出。
总结
以上就是这篇文章的全部内容了，希望本文的内容对大家的学习或者工作具有一定的参考学习价值，如果有疑问大家可以留言交流，谢谢大家对脚本之家的支持。
您可能感兴趣的文章:
- Python使用Socket(Https)Post登录百度的实现代码
- Python中使用socket发送HTTP请求数据接收不完整问题解决方法
- python通过get,post方式发送http请求和接收http响应的方法
- python使用urllib2提交http post请求的方法
- python中使用urllib2获取http请求状态码的代码例子
- python发送HTTP请求的方法小结
- Python中http请求方法库汇总
- Python发送http请求解析返回json的实例
- Python socket编程实例详解
- Python Socket编程入门教程
