WEBVTT

0:00:03.660000 --> 0:00:06.140000
 Hello everyone and welcome.

0:00:06.140000 --> 0:00:10.700000
 In this video we're going to be
 taking a look at URL encoding.

0:00:10.700000 --> 0:00:16.440000
 All right, so by this point we've taken
 a look at encoding in really two

0:00:16.440000 --> 0:00:18.040000
 facets or aspects.

0:00:18.040000 --> 0:00:24.300000
 We've talked about the character set
 encoding and also the process of

0:00:24.300000 --> 0:00:30.380000
 performing, you know, or HTML encoding
 for lack of a better explanation.

0:00:30.380000 --> 0:00:35.260000
 And the key thing with those two types
 of encoding, specifically regarding

0:00:35.260000 --> 0:00:40.180000
 or around the web, is that they were
 dealing with information or data

0:00:40.180000 --> 0:00:47.000000
 being encoded, pretty much in terms
 of how they were to be rendered or

0:00:47.000000 --> 0:00:51.600000
 the preparation of the data
 before it sent out.

0:00:51.600000 --> 0:00:56.020000
 The other aspect or the other facet
 of encoding which I touched upon in

0:00:56.020000 --> 0:01:01.480000
 the introduction to this section is
 encoding of data or information in

0:01:01.480000 --> 0:01:05.500000
 transit. And that's where URL
 encoding comes into play.

0:01:05.500000 --> 0:01:09.080000
 Now by this point in this learning path,
 you should already be familiar

0:01:09.080000 --> 0:01:13.660000
 with what URL encoding is in
 terms of the actual process.

0:01:13.660000 --> 0:01:18.960000
 But you may not know why this is done
 or how this works in terms of the

0:01:18.960000 --> 0:01:22.700000
 actual encoding, whether it's done,
 you know, by the actual web server

0:01:22.700000 --> 0:01:24.440000
 or by the client.

0:01:24.440000 --> 0:01:29.180000
 So hopefully I'll be able to explain
 the process, why it's important and

0:01:29.180000 --> 0:01:33.620000
 how it relates or ties back into web
 application penetration testing and

0:01:33.620000 --> 0:01:35.960000
 why it's important for you to understand.


0:01:35.960000 --> 0:01:42.440000
 So to begin with, what is URL encoding
 in terms of, you know, how it works,

0:01:42.440000 --> 0:01:45.240000
 what it is and, you know,
 what the process is like.

0:01:45.240000 --> 0:01:50.020000
 So to begin with URL encoding, also
 known as percent encoding, you'll

0:01:50.020000 --> 0:01:54.300000
 typically see those two terms being
 used interchangeably is a process

0:01:54.300000 --> 0:01:58.660000
 that is used to encode special characters,
 reserved characters and non

0:01:58.660000 --> 0:02:03.700000
 ASCII characters into a format that
 is safe for transmission within URLs

0:02:03.700000 --> 0:02:12.300000
 or URIs. URL stands for Uniform Resource
 Locators and URI stands for Uniform

0:02:12.300000 --> 0:02:14.480000
 Resource Identifiers.

0:02:14.480000 --> 0:02:18.840000
 So URLs are used to identify resources
 on the internet and certain characters

0:02:18.840000 --> 0:02:24.760000
 within URLs or URIs have special meanings
 in URLs, which, you know, makes

0:02:24.760000 --> 0:02:29.320000
 it necessary to encode them in order
 to avoid ambiguity and errors.

0:02:29.320000 --> 0:02:35.100000
 So the key thing here is that URL encoding
 is somewhat very similar to

0:02:35.100000 --> 0:02:39.720000
 HTML encoding in that the reason it
 is typically done is to prevent any

0:02:39.720000 --> 0:02:43.780000
 unintended interpretation
 of specific characters.

0:02:43.780000 --> 0:02:48.060000
 So as we did in the previous video with
 HTML encoding, what we were trying

0:02:48.060000 --> 0:02:52.740000
 to prevent is trying to prevent the
 web browser from interpreting, you

0:02:52.740000 --> 0:02:56.760000
 know, specific special characters, like,
 you know, let's say an HTML tag

0:02:56.760000 --> 0:02:58.640000
 or a script tag.

0:02:58.640000 --> 0:03:03.820000
 Literally, and instead just display
 them or, you know, render them as

0:03:03.820000 --> 0:03:08.260000
 they are or as the web application
 developer intended it.

0:03:08.260000 --> 0:03:12.520000
 It also, you know, prevents web application
 developers from making mistakes

0:03:12.520000 --> 0:03:17.980000
 in the event they did include a, you know,
 for example, an HTML or JavaScript

0:03:17.980000 --> 0:03:22.900000
 script tag within, you know, a
 particular paragraph or header.

0:03:22.900000 --> 0:03:28.100000
 It prevents that from being interpreted
 literally by the actual web browser.

0:03:28.100000 --> 0:03:33.140000
 And it's pretty much the same
 with URLs and URL encoding.

0:03:33.140000 --> 0:03:39.920000
 URLs are URL encoding is specifically used
 in cases where specific parameters

0:03:39.920000 --> 0:03:43.520000
 and their values are being
 passed in the URL.

0:03:43.520000 --> 0:03:47.340000
 So think of a, you know, simple
 PHP web page or a login form.

0:03:47.340000 --> 0:03:52.860000
 You'll typically see that you'll
 have, you know, www.test.com.

0:03:52.860000 --> 0:03:57.420000
 And then you'd see forward slash login
.php question mark and then the

0:03:57.420000 --> 0:04:01.000000
 parameter name. So, you know, username
 equals to, and then the value of

0:04:01.000000 --> 0:04:05.500000
 username and then and or ampersand password
 equals to, and then the value

0:04:05.500000 --> 0:04:07.100000
 of the password.

0:04:07.100000 --> 0:04:11.000000
 And when values are being passed, parameter
 values are being passed in

0:04:11.000000 --> 0:04:15.780000
 the URL, as you already know, you know,
 with cases like cross-site scripting

0:04:15.780000 --> 0:04:19.640000
 and SQL injection, those are prime
 targets because they're, you know,

0:04:19.640000 --> 0:04:20.840000
 application inputs.

0:04:20.840000 --> 0:04:25.660000
 And what you want to ensure as a web application
 developer is that whatever

0:04:25.660000 --> 0:04:31.100000
 input is being passed is URL encoded
 so that whatever reaches the server

0:04:31.100000 --> 0:04:35.360000
 is in an inner format that essentially
 tells the web server that, hey,

0:04:35.360000 --> 0:04:38.340000
 this is not to be processed literally.

0:04:38.340000 --> 0:04:39.800000
 This is what was passed in.

0:04:39.800000 --> 0:04:44.180000
 And any inclusion of special characters
 are to be treated as special characters

0:04:44.180000 --> 0:04:50.920000
 and not, you know, for example, PHP
 or, you know, JavaScript language.

0:04:50.920000 --> 0:04:54.980000
 As part of the, you know, server side
 language or the client side language

0:04:54.980000 --> 0:04:59.240000
 in the event that that value or the
 values of the parameters are to be

0:04:59.240000 --> 0:05:02.820000
 reflected back to the actual front
 end or back to the client.

0:05:02.820000 --> 0:05:08.060000
 And again, from the perspective of a web
 app and tester, why is this important?

0:05:08.060000 --> 0:05:11.320000
 It's important that you understand
 how the web application works with

0:05:11.320000 --> 0:05:15.920000
 regards to again, you know, sending parameter
 values back to the web server.

0:05:15.920000 --> 0:05:21.880000
 And if you went performing your testing,
 identifying the presence of URL

0:05:21.880000 --> 0:05:25.820000
 encoding is very easy, but I'll also
 tell you, you know, what you can

0:05:25.820000 --> 0:05:29.760000
 and cannot do. And we'll, you know, sort
 of allow you to expand your testing

0:05:29.760000 --> 0:05:32.320000
 or to identify potential vulnerabilities.


0:05:32.320000 --> 0:05:38.320000
 So URL encoding replaces unsafe characters
 with a percentage sign or symbol

0:05:38.320000 --> 0:05:44.000000
 that is followed by two hexadecimal
 digits that represent the ASCII code

0:05:44.000000 --> 0:05:44.640000
 of the character.

0:05:44.640000 --> 0:05:49.960000
 So we already explored this in the first
 video on the charset.org website,

0:05:49.960000 --> 0:05:54.160000
 where I showed you that, you know,
 pretty much the format is very easy

0:05:54.160000 --> 0:06:00.520000
 to understand. You have the percentage
 symbol and then the actual typically

0:06:00.520000 --> 0:06:06.000000
 the hexadecimal, if it's UTF-8, the
 hexadecimal, you know, value or code

0:06:06.000000 --> 0:06:10.700000
 point, if you will, the binary, sorry,
 hexadecimal representation of that

0:06:10.700000 --> 0:06:12.240000
 particular character.

0:06:12.240000 --> 0:06:16.540000
 And what this does is it allows URLs
 to be properly interpreted by web

0:06:16.540000 --> 0:06:20.560000
 browsers and their network components,
 even server side components.

0:06:20.560000 --> 0:06:24.580000
 So URLs sent over the internet must
 contain characters in the range of

0:06:24.580000 --> 0:06:27.440000
 the US ASCII code character set.

0:06:27.440000 --> 0:06:32.720000
 And if unsafe characters are present in
 the URL, encoding them is required.

0:06:32.720000 --> 0:06:36.380000
 Now the encoding is important because
 it limits the characters to be used

0:06:36.380000 --> 0:06:38.940000
 in a URL to a subset of
 specific characters.

0:06:38.940000 --> 0:06:40.560000
 What are these subsets?

0:06:40.560000 --> 0:06:45.320000
 Well, we have the unreserved character
 set or subset and the reserved

0:06:45.320000 --> 0:06:50.620000
 character subset, which comprise of in
 the case of unreserved characters,

0:06:50.620000 --> 0:06:55.200000
 you know, the alphabet, so A to Z, both
 upper case, lower case, the decimals,

0:06:55.200000 --> 0:06:56.700000
 so, you know, 0 to 9.

0:06:56.700000 --> 0:07:01.540000
 And then of course symbols like the
 square, not the square brackets, but

0:07:01.540000 --> 0:07:07.500000
 more so the hyphen or the minus, the
 full stop, the underscore and the

0:07:07.500000 --> 0:07:12.300000
 tilde. For reserved characters, we have
 the colon forward slash question

0:07:12.300000 --> 0:07:15.180000
 mark, the pound symbol,
 so on and so forth.

0:07:15.180000 --> 0:07:18.980000
 So again, this will make sense when once
 we get into the actual demo here,

0:07:18.980000 --> 0:07:23.480000
 but other characters are encoded by
 the use of a percentage character,

0:07:23.480000 --> 0:07:27.860000
 which I've already highlighted, but
 more importantly, the point that I

0:07:27.860000 --> 0:07:32.060000
 wanted to highlight here is the inclusion
 of two hexadecimal digits.

0:07:32.060000 --> 0:07:35.880000
 Reserved characters must be encoded when
 they have no special role inside

0:07:35.880000 --> 0:07:41.720000
 the URL. Okay, and when you visit a site
 URL encoding, this is very important,

0:07:41.720000 --> 0:07:44.000000
 is performed automatically by a browser.

0:07:44.000000 --> 0:07:48.120000
 Typically modern web browsers will perform
 this automatically, you know,

0:07:48.120000 --> 0:07:51.520000
 as a security precaution, but there's
 a caveat there that you need to

0:07:51.520000 --> 0:07:58.660000
 be aware of. If you appear to be a security
 feature, URL encoding is not

0:07:58.660000 --> 0:08:00.840000
 and should not be treated as such.

0:08:00.840000 --> 0:08:04.260000
 It is only a method that is used to
 send data across the internet in a

0:08:04.260000 --> 0:08:08.560000
 format that can be interpreted, you know,
 that can be interpreted correctly

0:08:08.560000 --> 0:08:12.980000
 by either the web server or the actual
 web browser in the format, you

0:08:12.980000 --> 0:08:16.340000
 know, of the actual value, essentially
 maintaining the integrity of the

0:08:16.340000 --> 0:08:17.900000
 data that's being sent.

0:08:17.900000 --> 0:08:22.340000
 So that is, it is not interpreted in
 any other way apart from, you know,

0:08:22.340000 --> 0:08:27.280000
 the actual intended form or format of
 the input that was actually intended

0:08:27.280000 --> 0:08:29.200000
 by the web application developer.

0:08:29.200000 --> 0:08:34.760000
 So URL encoding is very useful because
 it can either, you know, lower

0:08:34.760000 --> 0:08:38.240000
 or enlarge the attack surface depending
 on the web application you're

0:08:38.240000 --> 0:08:42.840000
 testing. And generally speaking, as I've
 pointed out already, web browsers

0:08:42.840000 --> 0:08:46.980000
 and other client side components automatically
 perform URL encoding, and

0:08:46.980000 --> 0:08:51.100000
 if a server side script engine is present,
 it'll automatically perform

0:08:51.100000 --> 0:08:52.420000
 the URL decoding.

0:08:52.420000 --> 0:08:55.940000
 So the way this works is very,
 very simple to understand.

0:08:55.940000 --> 0:09:00.240000
 So firstly, URL encoding is, you generally
 speaking will not see a web

0:09:00.240000 --> 0:09:04.340000
 application do it manually either using
 JavaScript on the client side

0:09:04.340000 --> 0:09:07.500000
 or even PHP on the server side.

0:09:07.500000 --> 0:09:10.640000
 Because the web browsers typically do
 it, but I'll show you how to implement

0:09:10.640000 --> 0:09:14.080000
 it and I'll show you the security benefits
 of doing so in terms of blocking

0:09:14.080000 --> 0:09:17.460000
 attacks like cross-site scripting
 and SQL injection.

0:09:17.460000 --> 0:09:22.440000
 But remember, one data is sent back,
 or when data is being sent from the

0:09:22.440000 --> 0:09:27.340000
 client to the web server, it is still
 going to be in the URL encoded format.

0:09:27.340000 --> 0:09:31.560000
 And what is typically done at that
 point is one of two things.

0:09:31.560000 --> 0:09:36.740000
 Either it is processed or the server
 side engine understands that it is

0:09:36.740000 --> 0:09:40.640000
 URL encoded and performs the decoding
 and then does whatever it wants

0:09:40.640000 --> 0:09:44.240000
 to the data. And that's where the issue
 is with regards to vulnerabilities

0:09:44.240000 --> 0:09:51.560000
 like cross-site scripting
 and SQL injection.

0:09:51.560000 --> 0:09:56.020000
 And more importantly, it all depends
 on how the backend or the server

0:09:56.020000 --> 0:10:02.760000
 side, you know, backend is going to handle
 what has been sent and whether

0:10:02.760000 --> 0:10:06.960000
 or not it goes through the decoding process,
 whether it strips those special

0:10:06.960000 --> 0:10:10.900000
 characters. But that you'll typically
 see being implemented on the client

0:10:10.900000 --> 0:10:15.600000
 side where inclusion of any, you know,
 let's say special characters like

0:10:15.600000 --> 0:10:20.500000
 that could be potentially dangerous is
 a sanitized or prevented from being

0:10:20.500000 --> 0:10:25.680000
 sent. But the key thing that I want you
 to focus on here is how URL encoding

0:10:25.680000 --> 0:10:30.480000
 is used to ensure that whatever is sent
 from the client side reaches the

0:10:30.480000 --> 0:10:34.880000
 server side in the format that
 was essentially intended.

0:10:34.880000 --> 0:10:38.680000
 And again, this will make a lot of sense
 once we take a look at a practical

0:10:38.680000 --> 0:10:43.900000
 example. Now, to sort of finally close
 on this theoretical section of

0:10:43.900000 --> 0:10:48.740000
 this video, you know, different web
 browsers running different engines

0:10:48.740000 --> 0:10:51.740000
 will handle URL encoding differently.

0:10:51.740000 --> 0:10:55.240000
 So in this particular table, you have
 a set of, you know, common or popular

0:10:55.240000 --> 0:10:56.520000
 web browsers on your left.

0:10:56.520000 --> 0:11:01.000000
 And on the top here, the headers, you can
 see that there's different examples

0:11:01.000000 --> 0:11:06.240000
 of URLs or the inclusion of parameters
 and parameter values, which is

0:11:06.240000 --> 0:11:08.940000
 where URL encoding really
 comes into play.

0:11:08.940000 --> 0:11:10.660000
 So this is typically what you'll see.

0:11:10.660000 --> 0:11:14.200000
 You know, you'll see the actual resource
 of the page like index.html.

0:11:14.200000 --> 0:11:16.000000
 It could be index.php.

0:11:16.000000 --> 0:11:19.300000
 And then the inclusion of a parameter,
 in this case, the parameter being

0:11:19.300000 --> 0:11:21.020000
 arg, that's what it's called.

0:11:21.020000 --> 0:11:23.320000
 And then the value is passed
 using an equal sign.

0:11:23.320000 --> 0:11:25.260000
 So arg equals test.

0:11:25.260000 --> 0:11:30.020000
 In this particular case, no URL encoding
 is performed or required because

0:11:30.020000 --> 0:11:35.380000
 again, we are passing in ASCII, you know,
 characters or a word that falls

0:11:35.380000 --> 0:11:39.500000
 within the actual ASCII character set.

0:11:39.500000 --> 0:11:44.160000
 And you can see how different web browsers
 will automatically perform

0:11:44.160000 --> 0:11:50.680000
 URL encoding based on the actual, you
 know, the actual data that is sent

0:11:50.680000 --> 0:11:51.940000
 as a value of the parameter.

0:11:51.940000 --> 0:11:55.680000
 So in the second, in the second example,
 you can see the inclusion of

0:11:55.680000 --> 0:12:00.560000
 spaces. And here it sort of highlights
 how different web browsers will

0:12:00.560000 --> 0:12:04.120000
 pretty much handle or perform
 the URL encoding.

0:12:04.120000 --> 0:12:07.440000
 So in this case, you can see we have
 the percentage symbol and 20, which

0:12:07.440000 --> 0:12:10.940000
 is hexadecimal for that essentially
 represents null.

0:12:10.940000 --> 0:12:14.580000
 And again, there's also other examples
 that are much more realistic.

0:12:14.580000 --> 0:12:18.320000
 So for example, the inclusion of an
 H1 tag, if, you know, we were trying

0:12:18.320000 --> 0:12:21.840000
 to do, let's say, HTML injection,
 you can see how it's going to be.

0:12:21.840000 --> 0:12:25.080000
 And now that is automatically
 encoded by the web browser.

0:12:25.080000 --> 0:12:29.680000
 But I'll also be showing you how this
 can be again, this can be enforced

0:12:29.680000 --> 0:12:33.440000
 or can be done on the client side using
 JavaScript and also on the server

0:12:33.440000 --> 0:12:37.980000
 side using PHP. But I'll also show you
 where web application developers

0:12:37.980000 --> 0:12:42.460000
 go wrong because you may think of this
 as a security feature, but it really

0:12:42.460000 --> 0:12:45.460000
 is not. And again, you'll
 see why shortly.

0:12:45.460000 --> 0:12:50.280000
 Also, the fact that I haven't included
 other popular web browsers like

0:12:50.280000 --> 0:12:53.760000
 Vivaldi and Brave, for example,
 is not a mistake.

0:12:53.760000 --> 0:12:57.080000
 The reason they're not included is
 because they all fall under Chrome

0:12:57.080000 --> 0:13:01.860000
 because they utilize the Chromium
 Web Engine or the Chromium Engine.

0:13:01.860000 --> 0:13:06.700000
 And, you know, Chrome pretty much Chrome
 and all other Chromium-based

0:13:06.700000 --> 0:13:11.120000
 web browsers will have the
 same form of URL encoding.

0:13:11.120000 --> 0:13:15.780000
 So the key thing that I want you to
 note is that in certain cases, like

0:13:15.780000 --> 0:13:19.740000
 in the case of Internet Explorer, which
 is not used that much anymore,

0:13:19.740000 --> 0:13:22.120000
 it does not automatically perform it.

0:13:22.120000 --> 0:13:26.660000
 So previously, this URL encoding was
 enforced both on the client side

0:13:26.660000 --> 0:13:28.220000
 and the server side.

0:13:28.220000 --> 0:13:33.320000
 But now as a security precaution, that's
 generally what is contextualized

0:13:33.320000 --> 0:13:35.660000
 under. It is performed
 by modern web browsers.

0:13:35.660000 --> 0:13:39.380000
 So with that being said, I'm now going
 to switch over to a demo again,

0:13:39.380000 --> 0:13:42.540000
 not using a lab because you can
 do this within your own system.

0:13:42.540000 --> 0:13:46.120000
 If you want to learn more about this
 and I highly encourage you to follow

0:13:46.120000 --> 0:13:50.520000
 along. It's important that you know
 how code works because as a web app

0:13:50.520000 --> 0:13:54.660000
 pen tester, this gives you an insight
 as to where vulnerabilities usually

0:13:54.660000 --> 0:13:58.920000
 fall or what causes these vulnerabilities
 and more importantly, how to

0:13:58.920000 --> 0:14:01.120000
 identify them and how to test for them.

0:14:01.120000 --> 0:14:03.800000
 So I'm going to switch over
 into my Kali Linux system.

0:14:03.800000 --> 0:14:07.180000
 There's two examples that I want to highlight
 that will be sort of building

0:14:07.180000 --> 0:14:09.120000
 on as we progress.

0:14:09.120000 --> 0:14:12.760000
 But yeah, I'm just going to switch
 over and we'll pick up from there.