A-A+

Python3 url格式（转换headers、表单和urlencode数据为字典格式）

2019年03月20日 12:02 学习笔记暂无评论共4348字 (阅读2,271 views次)

【注意：此文章为博主原创文章！转载需注意，请带原文链接，至少也要是txt格式！】

最近在学习爬虫时经常要复制浏览器的headers和表单数据到Python进行操作，但是复制过来的IE的数据格式是对用制表符（'\t'）进行分隔，而Chrome复制过来的是用冒号（':'）分隔，不能够直接转为字典格式使用。为了方便以后编程就自己写了个小程序进行转换。
Python的标准库应该有类似的方法，但自己找不到，知道的朋友麻烦告知下。谢谢！

# -*- coding: utf-8 -*-
"""
@author: Cy
"""
def strtodict(inputstr,sep=':',linesep='\n'):
    #linesep为行分隔符标记，默认为换行符。
    #sep为内部分隔符标记，默认为冒号
    if linesep !='\n':
        inputstr=inputstr.replace(linesep,'\n')
    strlist=RemoveEmptyLineInList(inputstr.split('\n'))
    strdicts={}
    for line in strlist:
        line=line.split(sep)
        if sep==':':
            strdicts[line[0]]=':'.join(line[1:])
        else:
            strdicts[line[0]]=line[1]
    return strdicts


def RemoveEmptyLineInList(listObj):
    newList = []
    for val in listObj:
        if val :
            newList.append(val);
    return newList

还发现了urllib库的urllib.parse.unquote()可以将IE浏览器里的已经urlencode的地址转化为原始数据。

tmppostdata=urllib.parse.unquote(urlencodedata)
postdata=strtodict(tmppostdata,sep='=',linesep='&')

别人另外一种更简洁的代码：

dict([item.split('=') for item in url_encode_data.split('&')])

以上内容文章来源：https://www.jianshu.com/p/dba073d10793

-
-
-

这里还有一个比较不错方式，上面的方式有最简洁的，但是最简洁的并不完善，在某些方面。

from urllib import parse

aa = "provider=spo&guid=94e&files=%7B%22it"
param_dict = parse.parse_qs(aa)
print(param_dict)

##输出：{'provider': ['spo'], 'guid': ['94e'], 'files': ['{"it']}

`
`
上面的代码有个瑕疵，我个人仔细论证后发现，parse_qs函数会URL解码键和值，简单的说就是url解码，把+号变空格，有很多人说正常啊，本来就是啊，，，但是！！！但是在某些情况下我们是不需要解码的，解码反而导致了其他的很多很多问题，所以。这里我说出一个简单的解决方案吧。
首先需要你找到这个引用的库文件：E:\xxx\Python37\Lib\urllib\parse.py
然后找到如下代码，我修改的地方我都加了标注。

def parse_qs(qs, keep_blank_values=False, strict_parsing=False,
             encoding='utf-8', errors='replace', url_encode=True):    ###注意这里做了修改，增加了url_encode
    """Parse a query given as a string argument.
 
        Arguments:
 
        qs: percent-encoded query string to be parsed
 
        keep_blank_values: flag indicating whether blank values in
            percent-encoded queries should be treated as blank strings.
            A true value indicates that blanks should be retained as
            blank strings.  The default false value indicates that
            blank values are to be ignored and treated as if they were
            not included.
 
        strict_parsing: flag indicating what to do with parsing errors.
            If false (the default), errors are silently ignored.
            If true, errors raise a ValueError exception.
 
        encoding and errors: specify how to decode percent-encoded sequences
            into Unicode characters, as accepted by the bytes.decode() method.
 
        Returns a dictionary.
    """
    parsed_result = {}
    pairs = parse_qsl(qs, keep_blank_values, strict_parsing,
                      encoding=encoding, errors=errors, url_encode=url_encode )    ###注意这里做了修改，增加了url_encode
    for name, value in pairs:
        if name in parsed_result:
            parsed_result[name].append(value)
        else:
            parsed_result[name] = [value]
    return parsed_result
 
 
def parse_qsl(qs, keep_blank_values=False, strict_parsing=True,
              encoding='utf-8', errors='replace', url_encode=True):    ###注意这里做了修改，增加了url_encode
    """Parse a query given as a string argument.
 
        Arguments:
 
        qs: percent-encoded query string to be parsed
D
        keep_blank_values: flag indicating whether blank values in
            percent-encoded queries should be treated as blank strings.
            A true value indicates that blanks should be retained as blank
            strings.  The default false value indicates that blank values
            are to be ignored and treated as if they were  not included.
 
        strict_parsing: flag indicating what to do with parsing errors. If
            false (the default), errors are silently ignored. If true,
            errors raise a ValueError exception.
 
        encoding and errors: specify how to decode percent-encoded sequences
            into Unicode characters, as accepted by the bytes.decode() method.
 
        Returns a list, as G-d intended.
    """
    qs, _coerce_result = _coerce_args(qs)
    pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')]
    r = []
    for name_value in pairs:
        if not name_value and not strict_parsing:
            continue
        nv = name_value.split('=', 1)
        if len(nv) != 2:
            if strict_parsing:
                raise ValueError("bad query field: %r" % (name_value,))
            # Handle case of a control-name with no equal sign
            if keep_blank_values:
                nv.append('')
            else:
                continue
        if len(nv[1]) or keep_blank_values:
            name = nv[0]        ###注意这里做了修改
            if url_encode:    ###注意这里做了修改
                name = name.replace('+', ' ')    ###注意这里做了修改
                name = unquote(name, encoding=encoding, errors=errors)    ###注意这里做了修改
            name = _coerce_result(name)
            value = nv[1]        ###注意这里做了修改
            if url_encode:    ###注意这里做了修改
                value = value.replace('+', ' ')    ###注意这里做了修改
                value = unquote(value, encoding=encoding, errors=errors)    ###注意这里做了修改
            value = _coerce_result(value)
            r.append((name, value))
    return r

def parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', url_encode=True): ###注意这里做了修改，增加了url_encode """Parse a query given as a string argument. Arguments: qs: percent-encoded query string to be parsed keep_blank_values: flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. strict_parsing: flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a ValueError exception. encoding and errors: specify how to decode percent-encoded sequences into Unicode characters, as accepted by the bytes.decode() method. Returns a dictionary. """ parsed_result = {} pairs = parse_qsl(qs, keep_blank_values, strict_parsing, encoding=encoding, errors=errors, url_encode=url_encode ) ###注意这里做了修改，增加了url_encode for name, value in pairs: if name in parsed_result: parsed_result[name].append(value) else: parsed_result[name] = [value] return parsed_result def parse_qsl(qs, keep_blank_values=False, strict_parsing=True, encoding='utf-8', errors='replace', url_encode=True): ###注意这里做了修改，增加了url_encode """Parse a query given as a string argument. Arguments: qs: percent-encoded query string to be parsed D keep_blank_values: flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. strict_parsing: flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a ValueError exception. encoding and errors: specify how to decode percent-encoded sequences into Unicode characters, as accepted by the bytes.decode() method. Returns a list, as G-d intended. """ qs, _coerce_result = _coerce_args(qs) pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')] r = [] for name_value in pairs: if not name_value and not strict_parsing: continue nv = name_value.split('=', 1) if len(nv) != 2: if strict_parsing: raise ValueError("bad query field: %r" % (name_value,)) # Handle case of a control-name with no equal sign if keep_blank_values: nv.append('') else: continue if len(nv[1]) or keep_blank_values: name = nv[0] ###注意这里做了修改 if url_encode: ###注意这里做了修改 name = name.replace('+', ' ') ###注意这里做了修改 name = unquote(name, encoding=encoding, errors=errors) ###注意这里做了修改 name = _coerce_result(name) value = nv[1] ###注意这里做了修改 if url_encode: ###注意这里做了修改 value = value.replace('+', ' ') ###注意这里做了修改 value = unquote(value, encoding=encoding, errors=errors) ###注意这里做了修改 value = _coerce_result(value) r.append((name, value)) return r

做了以上的修改后，你再次调用

from urllib import parse

aa = "provider=spo&guid=94e&files=%7B%22it"
param_dict = parse.parse_qs(aa, urlencode=False)  ##注意这里的调用
print(param_dict)

##输出：{'provider': ['spo'], 'guid': ['94e'], 'files': ['%7B%22it']}  ###注意和上面的区别，就是没有进行url解码而已哦。
#不过这里结果是list，如果想变为就是值的话，可以通过下面的方法。

for name, value in param_dict.items():
    if isinstance(value, list):
        param_dict[name] = ''.join(value)  ###注意前面两个单引号之间你可以用任意分隔符，也可以不用，直接连接。
print(param_dict)

##输出：{'provider': 'spo', 'guid': '94e', 'files': '%7B%22it'}

蜗居

窄小蜗居，虽非富贵王侯宅；清闲螺径，也异寻常百姓家。 woj → 蜗居

Python3 url格式（转换headers、表单和urlencode数据为字典格式）

给我留言取消回复

布施恩德可便相知重

微信扫一扫打赏

支付宝扫一扫打赏

给我留言取消回复