summaryrefslogtreecommitdiff
path: root/lib/libfetch/http.c
AgeCommit message (Collapse)Author
2023-08-16Remove $FreeBSD$: one-line .c patternWarner Losh
Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
2022-11-15libfetch: Pass a zeroed digest to DigestCalcResponse.John Baldwin
GCC 12 warns that passing "" (a constant of char[1]) to a parameter of type char[33] could potentially overread. It is not clear from the context that c->qops can never be "auth-int" (and if it can't, then the "auth-int" handling in DigestCalcResponse is dead code that should be removed since this is the only place the function is called). Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D36825
2022-04-20libfetch: remove a set-but-not-uswed variableStefan Eßer
2021-09-09fetch: do not confuse capacity and lengthBaptiste Daroussin
The patch converting fetch to getline (ee3ca711a898cf41330c320826ea1e0e6e451f1d), did confuse the capacity of the line buffer with the actual len of the read line confusing fetch -v.
2021-08-17libfetch: use more portable getline() interfaceDaniel Kolesa
this is for better portability in order to avoid using a function which is BSD-only or available via libbsd MFC after: 3 weeks
2021-04-01libfetch: Retry with proxy auth when server returns 407Renato Botelho
PR: 220468 Submitted by: Egil Hasting <egil.hasting@higen.org> (based on) Reviewed by: kevans, kp Approved by: kp MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D29533
2019-08-28Document fetchReqHTTP().Mark Johnston
Submitted by: Farhan Khan <khanzf@gmail.com> Reviewed by: 0mp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D18788 Notes: svn path=/head/; revision=351573
2018-11-27When deciding whether to send the complete URL or just the document part,Dag-Erling Smørgrav
we were looking at the original URL rather than the one we were currently processing. This meant that if we were trying to retrieve an HTTP URL but were redirected to an HTTPS URL, and HTTPS proxying was enabled, we would send an invalid request and most likely get garbage back. MFC after: 3 days Notes: svn path=/head/; revision=341072
2018-11-27A few more cases where strcasecmp() is no longer required.Dag-Erling Smørgrav
MFC after: 1 week Notes: svn path=/head/; revision=341014
2018-05-29Fix a few (but far from all) style issues.Dag-Erling Smørgrav
MFC after: 3 weeks Notes: svn path=/head/; revision=334319
2018-05-29Use __VA_ARGS__ to simplify the DEBUG macro.Dag-Erling Smørgrav
MFC after: 3 weeks Notes: svn path=/head/; revision=334317
2018-05-12Preserve if-modified-since timestamps across redirects.Dag-Erling Smørgrav
PR: 224426 MFC after: 1 week Notes: svn path=/head/; revision=333571
2017-11-26lib: further adoption of SPDX licensing ID tags.Pedro F. Giffuni
Mainly focus on files that use BSD 2-Clause license, however the tool I was using mis-identified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326219
2017-03-17r308996 broke IP literals by assuming that a colon could only occur asDag-Erling Smørgrav
a separator between host and port, and using strchr() to search for it. Rewrite fetch_resolve() so it handles bracketed literals correctly, and remove similar code elsewhere to avoid passing unbracketed literals to fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals correctly even if we do not have the ability to connect to them. While there, fix an off-by-one error which caused HTTP 400 errors to be misinterpreted as redirects. PR: 217723 MFC after: 1 week Reported by: bapt, bz, cem, ngie Notes: svn path=/head/; revision=315455
2017-03-05Fix partial requests (used by fetch -r) when the requested file isDag-Erling Smørgrav
already complete. Since 416 is an error code, any Content-Range header in the response would refer to the error message, not the requested document, so relying on the value of size when we know we got a 416 is wrong. Instead, just verify that offset == 0 and assume that we've reached the end of the document (if offset > 0, we did not request a range, and the server is screwing with us). Note that we cannot distinguish between reaching the end and going past it, but that is a flaw in the protocol, not in the code, so we just have to assume that the caller knows what it's doing. A smart caller would request an offset slightly before what it believes is the end and compare the result to what is already in the file. PR: 212065 Reported by: mandree MFC after: 3 weeks Notes: svn path=/head/; revision=314701
2016-12-30Fix inverted loop condition which broke multi-line responses to CONNECT.Dag-Erling Smørgrav
PR: 194483 Submitted by: Miłosz Kaniewski <milosz.kaniewski@gmail.com> MFC after: 1 week Notes: svn path=/head/; revision=310823
2016-05-31r169386 (PR 112515) was incomplete: it treated 307 as an error exceptDag-Erling Smørgrav
in verbose mode, and did not handle 308 at all. r241840 (PR 172451) added support for 308, but with the same bug. Correctly handle both by recognizing them as redirects in all places where we check the HTTP result code. PR: 112515 173451 209546 Submitted by: novel@ MFC after: 1 week Notes: svn path=/head/; revision=301027
2016-05-12Use strlcpy() instead of strncpy() to copy the string returned byDon Lewis
setlocale() so that static analyzers know that the string is NUL terminated. This was causing a false positive in Coverity even though the longest string returned by setlocale() is ENCODING_LEN (31) and we are copying into a 64 byte buffer. This change is also a bit of an optimization since we don't need the strncpy() feature of padding the rest of the destination buffer with NUL characters. Reported by: Coverity CID: 974654 Notes: svn path=/head/; revision=299520
2016-02-11Fix double-free error: r289419 moved all error handling in http_connect()Dag-Erling Smørgrav
to the end of the function, but did not remove a fetch_close() call which was made redundant by the one in the shared error-handling code. PR: 206774 Submitted by: Christian Heckendorf <heckendorfc@gmail.com> MFC after: 3 days Notes: svn path=/head/; revision=295536
2015-12-16As a followup to r292330, standardize on size_t and add a few comments.Dag-Erling Smørgrav
Notes: svn path=/head/; revision=292332
2015-12-16Reset bufpos to 0 immediately after refilling the buffer. Otherwise, weDag-Erling Smørgrav
risk leaving the connection in an indeterminate state if the server fails to send a chunk delimiter. Depending on the application and on the sizes of the preceding chunks, the result can be anything from missing data to a segfault. With this patch, it will be reported as a protocol error. PR: 204771 MFC after: 1 week Notes: svn path=/head/; revision=292330
2015-11-29Fix buildworld after r291453, similar to r284346: url->user and url->pwdDimitry Andric
are arrays, so they can never be NULL. Reported by: many Pointy hat to: des Notes: svn path=/head/; revision=291461
2015-11-29Use .netrc for HTTP sites and proxies, not just FTP.Dag-Erling Smørgrav
PR: 193740 Submitted by: TEUBEL György <tgyurci@gmail.com> MFC after: 1 week Notes: svn path=/head/; revision=291453
2015-10-16Fix two bugs in HTTPS tunnelling:Dag-Erling Smørgrav
- If the proxy returns a non-200 result, set the error code accordingly so the caller / user gets a somewhat meaningful error message. - Consume and discard any HTTP response header following the result line. PR: 194483 Tested by: Fabian Keil <fk@fabiankeil.de> MFC after: 1 week Notes: svn path=/head/; revision=289419
2015-07-04Remove unused variable to silence clang warning.Marcelo Araujo
Differential Revision: D2683 Reviewed by: rodrigc, bapt Notes: svn path=/head/; revision=285141
2015-06-13Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:Dimitry Andric
lib/libfetch/http.c:1628:26: error: address of array 'purl->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = purl->user ? ~~~~~~^~~~ ~ lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = purl->pwd? ~~~~~~^~~~ lib/libfetch/http.c:1657:25: error: address of array 'url->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = url->user ? ~~~~~^~~~ ~ lib/libfetch/http.c:1659:29: error: address of array 'url->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = url->pwd ? ~~~~~^~~ ~ lib/libfetch/http.c:1669:25: error: address of array 'url->user' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.user = url->user ? ~~~~~^~~~ ~ lib/libfetch/http.c:1671:29: error: address of array 'url->pwd' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion] aparams.password = url->pwd ? ~~~~~^~~ ~ Since url->user and url->pwd are arrays, they can never be NULL, so the checks can be removed. Reviewed by: bapt MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D2673 Notes: svn path=/head/; revision=284346
2014-06-05Add support for arbitrary http requestsBaptiste Daroussin
Submitted by: Alex Hornung <alex@alexhornung.com> Reviewed by: des Obtained from: Dragonfly MFC after: 3 week Notes: svn path=/head/; revision=267133
2014-06-05Remove unnecessary semicolonsBaptiste Daroussin
Patch by Sascha Wildner <saw@online.de> for Dragonfly Reviewed by: des Obtained from: Dragonfly MFC after: 1 week Notes: svn path=/head/; revision=267132
2014-06-05Use NULL instead of 0Baptiste Daroussin
Patch by Sascha Wildner <saw@online.de> for Dragonfly Reviewed by: des Obtained from: Dragonfly MFC after: 1 week Notes: svn path=/head/; revision=267131
2014-06-05If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.Dag-Erling Smørgrav
PR: 184507 Submitted by: jbeich@tormail.org (with modifications) MFC after: 1 week Notes: svn path=/head/; revision=267127
2014-03-11Support Last-Modified behind proxies which return UTC instead of GMT.Bryan Drewery
The standard states that GMT must be used, but that UTC is equivalent. Still parse UTC as otherwise this causes problems for pkg(8). It will refetch the repository every time 'pkg update' or other remote operations are used behind these proxies. RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal to UTC (Coordinated Universal Time)."" Approved by: bapt (mentor) Reviewed by: des, peter Sponsored by: EMC / Isilon Storage Division MFC after: 1 week Notes: svn path=/head/; revision=263021
2014-01-30Bump copyright datesDag-Erling Smørgrav
Notes: svn path=/head/; revision=261284
2014-01-29r261230 broke the cases where the amount of data to be read is notDag-Erling Smørgrav
known in advance, or where the caller doesn't care and just keeps reading until it hits EOF. In fetch_read(): the socket is non-blocking, so read() will return 0 on EOF, and -1 (errno == EAGAIN) when the connection is still open but there is no data waiting. In the first case, we should immediately return 0. The EINTR case was also broken, although not in a way that matters. In fetch_writev(): use timersub() and timercmp() as in fetch_read(). In http_fillbuf(): set errno to a sensible value when an invalid chunk header is encountered. In http_readfn(): as in fetch_read(), a zero return from down the stack indicates EOF, not an error. Furthermore, when io->error is EINTR, clear it (but no errno) before returning so the caller can retry after dealing with the interrupt. MFC after: 3 days Notes: svn path=/head/; revision=261263
2014-01-28Solve http buffering issues and hangs once and for all (hopefully!) byDag-Erling Smørgrav
simply not trying to return exactly what the caller asked for - just return whatever we got and let the caller be the judge of whether it was enough. If an error occurs or the connection times out after we already received some data, return a short read, under the assumption that the next call will fail or time out before we read anything. As it turns out, none of the code that calls fetch_read() assumes an all-or-nothing result anyway, except for a couple of lines where we read the CR LF at the end of a hunk in HTTP hunked encoding, so the changes outside of fetch_read() and http_readfn() are minimal. While there, replace select(2) with poll(2). MFC after: 3 days Notes: svn path=/head/; revision=261230
2013-08-22Even though it doesn't really make sense in the context of a CONNECTDag-Erling Smørgrav
request, RFC 2616 14.23 mandates the presence of the Host: header in all HTTP 1.1 requests. PR: kern/181445 Submitted by: Kimo <kimor79@yahoo.com> MFC after: 3 days Notes: svn path=/head/; revision=254650
2013-07-30Include an Accept header in requests.Dag-Erling Smørgrav
PR: kern/180917 MFC after: 1 week Notes: svn path=/head/; revision=253805
2013-07-26Implement certificate verification, and many other SSL-relatedDag-Erling Smørgrav
imrovements; complete details in the PR. PR: kern/175514 Submitted by: Michael Gmelin <freebsd@grem.de> MFC after: 1 week Notes: svn path=/head/; revision=253680
2013-07-21Use the correct request syntax for proxied (tunneled) HTTPS requests.Dag-Erling Smørgrav
PR: bin/180666 MFC after: 3 days Notes: svn path=/head/; revision=253514
2013-04-12Use the CONNECT method to proxy HTTPS connections through HTTP proxies.Dag-Erling Smørgrav
PR: bin/80176 Submitted by: Yuichiro NAITO <naito.yuichiro@gmail.com> Notes: svn path=/head/; revision=249431
2012-11-16Fix weird indentation.Dag-Erling Smørgrav
Notes: svn path=/head/; revision=243149
2012-10-22Implement HTTP 305 redirect handling.Eitan Adler
PR: 172452 Submitted by: gcooper Reviewed by: des Approved by: cperciva MFC after: 1 week Notes: svn path=/head/; revision=241841
2012-10-22Don't deny non-temporary redirects if the -A option is set (perEitan Adler
the man page) [0] While here add support for draft-reschke-http-status-308-07 PR: 172451 [0] Submitted by: gcooper [0] Reviewed by: des Approved by: cperciva MFC after: 1 week Notes: svn path=/head/; revision=241840
2012-10-22Be a bit more lenient in the maximum number of redirects allowed.Eitan Adler
Chrome and Firefox have a limit of 20. IE has a limit of 8. Reviewed by: des Approved by: cperciva MFC after: 3 days Notes: svn path=/head/; revision=241839
2012-09-14Use libmd if and only if OpenSSL is not available.Dag-Erling Smørgrav
PR: bin/171402 MFC after: 3 days Notes: svn path=/head/; revision=240496
2012-04-30Don't reuse credentials if redirected to a different host.Dag-Erling Smørgrav
Submitted by: Niels Heinen <heinenn@google.com> MFC after: 3 weeks Notes: svn path=/head/; revision=234838
2012-01-18Fix two issues related to the use of SIGINFO in fetch(1) to displayDag-Erling Smørgrav
progress information. The first is that fetch_read() (used in the HTTP code but not the FTP code) can enter an infinite loop if it has previously been interrupted by a signal. The second is that when it is interrupted, fetch_read() will discard any data it may have read up to that point. Luckily, both bugs are extremely timing-sensitive and therefore difficult to trigger. PR: bin/153240 Submitted by: Mark <markjdb@gmail.com> MFC after: 3 weeks Notes: svn path=/head/; revision=230307
2011-10-19latin1 -> utf8Dag-Erling Smørgrav
Notes: svn path=/head/; revision=226537
2011-09-27Update copyright dates and strip my middle name.Dag-Erling Smørgrav
Notes: svn path=/head/; revision=225814
2011-05-12Increase WARNS to 4.Dag-Erling Smørgrav
Notes: svn path=/head/; revision=221822
2011-05-12Mechanical whitespace cleanup.Dag-Erling Smørgrav
Notes: svn path=/head/; revision=221821
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-22 15:31:39 +0000