+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Programming (https://www.sickgaming.net/forum-76.html)
+--- Forum: Python (https://www.sickgaming.net/forum-83.html)
+--- Thread: [Tut] [Collection] URL Decoding Methods (/thread-96836.html)
[Tut] [Collection] URL Decoding Methods - xSicKxBot - 08-22-2020
[Collection] URL Decoding Methods
<div><p><a href="https://en.wikipedia.org/wiki/Percent-encoding" target="_blank" rel="noreferrer noopener" title="https://en.wikipedia.org/wiki/Percent-encoding">URL encoding</a><em> “is a method to <a href="https://en.wikipedia.org/wiki/Binary-to-text_encoding">encode</a> information in a <a href="https://en.wikipedia.org/wiki/Uniform_Resource_Identifier">Uniform Resource Identifier</a> (URI)”</em>. It is also called <em>Percent-encoding</em> because percentage symbols are used to encode certain reserved characters:</p>
<figure class="wp-block-table is-style-stripes">
<table>
<tbody>
<tr>
<td><code>!</code></td>
<td><code>#</code></td>
<td><code>$</code></td>
<td><code>%</code></td>
<td><code>&</code></td>
<td><code>'</code></td>
<td><code>(</code></td>
<td><code>)</code></td>
<td><code>*</code></td>
<td><code>+</code></td>
<td><code>,</code></td>
<td><code>/</code></td>
<td><code>:</code></td>
<td><code>;</code></td>
<td><code>=</code></td>
<td><code>?</code></td>
<td><code>@</code></td>
<td><code>[</code></td>
<td><code>]</code></td>
</tr>
<tr>
<td><code>%21</code></td>
<td><code>%23</code></td>
<td><code>%24</code></td>
<td><code>%25</code></td>
<td><code>%26</code></td>
<td><code>%27</code></td>
<td><code>%28</code></td>
<td><code>%29</code></td>
<td><code>%2A</code></td>
<td><code>%2B</code></td>
<td><code>%2C</code></td>
<td><code>%2F</code></td>
<td><code>%3A</code></td>
<td><code>%3B</code></td>
<td><code>%3D</code></td>
<td><code>%3F</code></td>
<td><code>%40</code></td>
<td><code>%5B</code></td>
<td><code>%5D</code></td>
</tr>
</tbody>
</table>
</figure>
<p>This article collects various ways to <strong><em>decode </em></strong>an URL encoded string. Let’s get started!</p>
<h2>Python 2</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ alias urldecode='python -c "import sys, urllib as ul; \ print ul.unquote_plus(sys.argv[1])"' $ alias urlencode='python -c "import sys, urllib as ul; \ print ul.quote_plus(sys.argv[1])"'</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener" title="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding">Source</a></p>
<p>Here’s an example usage:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ urldecode 'q+werty%3D%2F%3B'
q werty=/; $ urlencode 'q werty=/;'
q+werty%3D%2F%3B</pre>
<h2>Python 3</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ alias urldecode='python3 -c "import sys, urllib.parse as ul; \ print(ul.unquote_plus(sys.argv[1]))"' $ alias urlencode='python3 -c "import sys, urllib.parse as ul; \ print (ul.quote_plus(sys.argv[1]))"'</pre>
<p>Here’s an example usage:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ urldecode 'q+werty%3D%2F%3B'
q werty=/; $ urlencode 'q werty=/;'
q+werty%3D%2F%3B</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>sed</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ sed 's@+@ @g;s@%@\\x@g' file | xargs -0 printf "%b"</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>sed with echo -e</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ sed -e's/%\([0-9A-F][0-9A-F]\)/\\\\\x\1/g' file | xargs echo -e</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>sed with alias</h2>
<p>For convenience, you may want to use an alias:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ alias urldecode='sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b"'</pre>
<p>If you want to decode, you can now simply use:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ echo "http%3A%2F%2Fwww" | urldecode
http://www</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>Bash</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">input="http%3A%2F%2Fwww"
decoded=$(printf '%b' "${input//%/\\x}")</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<p>To handle pluses (<code>+</code>) correctly, replace them with spaces using <code>sed</code>:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">decoded=$(input=${input//+/ }; printf "${input//%/\\x}")</pre>
<h2>Bash + urlencode() + urldecode() Functions</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">urlencode() { # urlencode <string> local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf '%%%02X' "'$c" ;; esac done
} urldecode() { # urldecode <string> local url_encoded="${1//+/ }" printf '%b' "${url_encoded//%/\\x}"
}</pre>
<p>Sources:</p>
<ul>
<li><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">StackOverflow</a></li>
<li><a href="https://github.com/sixarm/urldecode.sh" target="_blank" rel="noreferrer noopener">https://github.com/sixarm/urldecode.sh</a></li>
</ul>
<h2>bash + xxd</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">urlencode() { local length="${#1}" for (( i = 0; i < length; i++ )); do local c="${1:i:1}" case $c in [a-zA-Z0-9.~_-]) printf "$c" ;; *) printf "$c" | xxd -p -c1 | while read x;do printf "%%%s" "$x";done esac
done
}</pre>
<p>Sources:</p>
<ul>
<li><a href="https://gist.github.com/cdown/1163649">cdown’s gist</a></li>
<li><a href="https://stackoverflow.com/questions/6250698/shell-script-to-urldecode-file-contents">Stackoverflow</a></li>
<li><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Another Source</a></li>
</ul>
<h2>PHP</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ echo oil+and+gas | php -r 'echo urldecode(fgets(STDIN));' // Or: php://stdin
oil and gas</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>PHP Library</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">php -r 'echo urldecode("oil+and+gas");'</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>Perl</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h2>Perl to Process File</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file</pre>
<p><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></p>
<h3>awk</h3>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..</pre>
<p>Sources:</p>
<ul>
<li><a href="https://stackoverflow.com/a/14000368/55075" target="_blank" rel="noreferrer noopener">anon</a>‘s comment in StackOverflow article</li>
<li><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Source</a></li>
<li><a href="https://stackoverflow.com/questions/3728049/using-awk-printf-to-urldecode-text">Using awk printf to urldecode text</a>.</li>
</ul>
<h2>Python 2 urllib.unquote</h2>
<p>The <a href="http://docs.python.org/2.7/library/urllib.html#urllib.unquote"><code>urllib.unquote</code></a> is a special function in Python’s built-in standard library <code>urllib</code> that does what you need:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">decoded_url=$(python2 -c 'import sys, urllib; print urllib.unquote(sys.argv[1])' "$encoded_url")</pre>
<p>You can also use it to modify a file:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">python2 -c 'import sys, urllib; print urllib.unquote(sys.stdin.read())' <file >file.new &&
mv -f file.new file</pre>
<p>Source: <a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding</a></p>
<h2>Python 3 urllib.parse.unquote</h2>
<p>If you run Python 3 on your system (like most people would), use the alternative function <a href="http://docs.python.org/3.1/library/urllib.parse.html#urllib.parse.unquote" target="_blank" rel="noreferrer noopener"><code>urllib.parse.unquote</code></a>. To check your version, visit <a href="https://blog.finxter.com/how-to-check-your-python-version/" target="_blank" rel="noreferrer noopener" title="How to Check Your Python Version? A Helpful Guide">this article</a>.</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">decoded_url=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))' "$encoded_url")</pre>
<p>Again, you can use the function to process a file as follows:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">python3 -c 'import sys, urllib; print(urllib.parse.unquote(sys.stdin.read()))' <file >file.new &&
mv -f file.new file</pre>
<p>Source: <a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding</a></p>
<h2>Perl URI::Escape</h2>
<p>The <a href="http://search.cpan.org/perldoc?URI%3A%3AEscape"><code>URI::Escape</code></a> solves the problem of URL decoding for Perl users. </p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")</pre>
<p>You can use the function to process a file as follows:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file</pre>
<p>Source: <a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding</a></p>
<h2>Perl One-Liner Without Installing Modules</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ perl -pe 's/\%(\w\w)/chr hex $1/ge'</pre>
<p>Here’s a usage example:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ echo '%21%22' | perl -pe 's/\%(\w\w)/chr hex $1/ge'
!"</pre>
<p>Source: <a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding</a></p>
<h2>Bash Regex</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }</pre>
<p>Now, you can use the function as a command like this:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ urldecode https%3A%2F%2Fgoogle.com%2Fsearch%3Fq%3Durldecode%2Bbash
https://google.com/search?q=urldecode+bash</pre>
<p>If you need to assign some variables, use this strategy:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">$ x="http%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3Durldecode%2Bbash"
$ y=$(urldecode "$x")
$ echo "$y"
http://stackoverflow.com/search?q=urldecode+bash</pre>
<p>Source: <a href="https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell" target="_blank" rel="noreferrer noopener">https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell</a></p>
<h2>GNU Awk</h2>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">#!/usr/bin/awk -fn
@include "ord"
BEGIN { RS = "%.."
}
{ printf "%s", $0 if (RT != "") { printf "%s", chr("0x" substr(RT, 2)) }
}</pre>
<p>Source: <a href="https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell" target="_blank" rel="noreferrer noopener">https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell</a></p>
</p>
<h3>References</h3>
<ul>
<li><a href="https://web.archive.org/web/20110111070924/http://www.ruslanspivak.com:80/2010/06/02/urlencode-and-urldecode-from-a-command-line/" target="_blank" rel="noreferrer noopener">Urlencode and urldecode from a command line</a></li>
<li><a href="https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding" target="_blank" rel="noreferrer noopener">Decoding URL encoding (percent encoding)</a></li>
<li><a href="https://stackoverflow.com/questions/6250698/how-to-decode-url-encoded-string-in-shell">How to decode URL-encoded string in shell?</a></li>
<li><a href="https://gist.github.com/cdown/1163649" target="_blank" rel="noreferrer noopener">https://gist.github.com/cdown/1163649 </a></li>
<li><a href="https://dev.to/k4ml/python-urldecode-on-command-line-2ek9" target="_blank" rel="noreferrer noopener" title="https://dev.to/k4ml/python-urldecode-on-command-line-2ek9">Python urldecode on command line</a></li>
</ul>
</div>
https://www.sickgaming.net/blog/2020/08/22/collection-url-decoding-methods/