summaryrefslogtreecommitdiff
path: root/lib/libfetch
AgeCommit message (Collapse)Author
2025-12-16libfetch: Fix -Wunterminated-string-initializationAlex Richardson
This defaults to an error in clang HEAD, use a char-by-char initializer instead. Reviewed by: emaste, jhb MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D52532
2025-07-17lib: Fix calls that naively set F_SETFD.Ricardo Branco
With the recent inclusion of the FD_CLOFORK and FD_RESOLVE_BENEATH flags, we must avoid clearing them when setting only FD_CLOEXEC. Signed-off-by: Ricardo Branco <rbranco@suse.de> Reviewed by: kib, markj MFC after: 1 month Pull Request: https://github.com/freebsd/freebsd-src/pull/1766
2024-08-21libfetch: don't include fragments in HTTP requestsPietro Cerutti
Summary: Fragments are reserved for client-side processing, see https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 Also, some servers don't like to receive HTTP requests with fragments. ``` $ fetch 'https://dropbox.com/a/b' fetch: https://dropbox.com/a/b: Not Found $ fetch 'https://dropbox.com/a/b#' fetch: https://dropbox.com/a/b#: Bad Request ``` This is a real-world scenario, where some download link from dropbox (eventually) redirects to an URL with a fragment: ``` $ fetch -v 'https://www.dropbox.com/sh/<some>/<thing>?dl=1' 2>&1 | grep requesting requesting https://www.dropbox.com/sh/<some>/<thing>?dl=1 requesting https://www.dropbox.com/scl/fo/<foo>/<bar>?rlkey=<baz>&dl=1 requesting https://<boo>.dl.dropboxusercontent.com/zip_download_get/<some-long-strig># ``` See how the last redirect ends with a `#`. Currently, libfetch includes the ending fragment and makes it impossible to download the file. Differential Revision: https://reviews.freebsd.org/D46318 MFC after: 2 weeks
2024-07-15Remove residual blank line at start of MakefileWarner Losh
This is a residual of the $FreeBSD$ removal. MFC After: 3 days (though I'll just run the command on the branches) Sponsored by: Netflix
2024-07-11libfetch: parse IPv4address in IPv6address properlyKa Ho Ng
This fix parsing problems of IPv6 addresses which contains IPv4 addresses. One example is ::192.168.0.1. Documents are in below: https://datatracker.ietf.org/doc/html/rfc2373#page-22 Sponsored by: Juniper Networks, Inc. MFC after: 1 week Reviewed by: jhibbits Differential Revision: https://reviews.freebsd.org/D45896
2024-03-25libfetch: parse scheme://domain:/ correctlyKa Ho Ng
This improves URL-parsing compability with cURL, and unbreaks parsing of similar kinds of URLs after commit 8d9de5b10a24. Sponsored by: Juniper Networks, Inc. Reviewed by: des MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D44493
2024-02-02lib/libfetch/common.c: remove an extra semicolonrilysh
Signed-off-by: rilysh <nightquick@proton.me> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/959
2023-11-26lib: Automated cleanup of cdefs and other formattingWarner Losh
Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/ Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/ Sponsored by: Netflix
2023-10-08libfetch, fetch: Stop recommending the use of ca_root_nss.Dag-Erling Smørgrav
MFC after: 3 days Reviewed by: kevans, emaste Differential Revision: https://reviews.freebsd.org/D42119
2023-10-03libfetch: don't rely on ca_root_nss for certificate validationMichael Osipov
Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059
2023-08-16Remove $FreeBSD$: two-line nroff patternWarner Losh
Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/
2023-08-16Remove $FreeBSD$: one-line sh patternWarner Losh
Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/
2023-08-16Remove $FreeBSD$: one-line .c patternWarner Losh
Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
2023-08-16Remove $FreeBSD$: two-line .h patternWarner Losh
Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
2023-06-24libfetch: remove all old OpenSSL supportEnji Cooper
This change removes pre-OpenSSL 1.1 supporting code and removes/adjusted preprocessor conditionals which were tautilogically true as FreeBSD main has shipped with OpenSSL 1.1+ for some time. Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D40711
2023-05-25libfetch: specify OpenSSL 1.1 APIsPierre Pronchery
OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in use for the purpose of hiding deprecated interfaces and enabling the appropriate deprecation notices. This change is a NFC while we're still using OpenSSL 1.1.1 but will avoid deprecation warnings upon the switch to OpenSSL 3.0. Future work should migrate to use the OpenSSL 3.0 APIs. PR: 271615 Reviewed by: emaste Event: Kitchener-Waterloo Hackathon 202305 Sponsored by: The FreeBSD Foundation
2023-05-25libfetch: do not call deprecated OpenSSL functionsEd Maste
As of OpenSSL 1.1 SSL_library_init() and SSL_load_error_strings() are deprecated. There are replacement initialization functions but they do not need to be called: "As of version 1.1.0 OpenSSL will automatically allocate all resources that it needs so no explicit initialisation is required." Wrap both calls in an OPENSSL_VERSION_NUMBER block. PR: 271615 Reviewed by: Pierre Pronchery <pierre@freebsdfoundation.org> Event: Kitchener-Waterloo Hackathon 202305 Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D40265
2022-11-15libfetch: Pass a zeroed digest to DigestCalcResponse.John Baldwin
GCC 12 warns that passing "" (a constant of char[1]) to a parameter of type char[33] could potentially overread. It is not clear from the context that c->qops can never be "auth-int" (and if it can't, then the "auth-int" handling in DigestCalcResponse is dead code that should be removed since this is the only place the function is called). Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D36825
2022-11-02fetch: support EAI_ADDRFAMILY error, correct two error messagesMike Karels
With the change to return EAI_ADDRFAMILY from getaddrinfo(), fetch would print "Unknown resolver error" for that error. Add that error and its string to libfetch's table, using an #ifdef just in case. Correct error strings for EAI_NODATA (although it is currently unused) and EAI_NONAME. Should maybe rework the code to use gai_strerror(3), but that doesn't map directly, and the current strings are shortened. Reviewed in https://reviews.freebsd.org/D37139 with related changes. Reviewed by: bz MFC after: 1 month
2022-10-03libfetch: Use memcpy in place of an odd strncpy.John Baldwin
The length passed to strncpy is the length of the source string, not the destination buffer. This triggers a non-fatal warning in GCC 12. Hoewver, the code is also odd. It is really just a memcpy of the string without its nul terminator. For that use case, memcpy is clearer. Reviewed by: imp, emaste Differential Revision: https://reviews.freebsd.org/D36824
2022-09-17There's no PEM(3) anywhere around; delete reference.Jens Schweikhardt
2022-04-20libfetch: remove a set-but-not-uswed variableStefan Eßer
2021-12-21pkgbase: Create a FreeBSD-fetch packageEmmanuel Vadot
It's useful for small image to fetch some data but we don't want to install utilities nor bloat runtime. MFC after: 2 weeks Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D33463
2021-09-09fetch: do not confuse capacity and lengthBaptiste Daroussin
The patch converting fetch to getline (ee3ca711a898cf41330c320826ea1e0e6e451f1d), did confuse the capacity of the line buffer with the actual len of the read line confusing fetch -v.
2021-08-24Fix libfetch out of bounds read.Gordon Tetlow
Approved by: so Security: SA-21:15.libfetch Security: CVE-2021-36159
2021-08-17libfetch: use more portable getline() interfaceDaniel Kolesa
this is for better portability in order to avoid using a function which is BSD-only or available via libbsd MFC after: 3 weeks
2021-04-01libfetch: Retry with proxy auth when server returns 407Renato Botelho
PR: 220468 Submitted by: Egil Hasting <egil.hasting@higen.org> (based on) Reviewed by: kevans, kp Approved by: kp MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D29533
2020-11-24Remove support for SSLv3 from fetch(3).Jung-uk Kim
Support for SSLv3 was already removed from OpenSSL (r361392). Differential Revision: https://reviews.freebsd.org/D24947 Notes: svn path=/head/; revision=368000
2020-10-27Replace literal uses of /usr/local in C sources with _PATH_LOCALBASEStefan Eßer
Literal references to /usr/local exist in a large number of files in the FreeBSD base system. Many are in contributed software, in configuration files, or in the documentation, but 19 uses have been identified in C source files or headers outside the contrib and sys/contrib directories. This commit makes it possible to set _PATH_LOCALBASE in paths.h to use a different prefix for locally installed software. In order to avoid changes to openssh source files, LOCALBASE is passed to the build via Makefiles under src/secure. While _PATH_LOCALBASE could have been used here, there is precedent in the construction of the path used to a xauth program which depends on the LOCALBASE value passed on the compiler command line to select a non-default directory. This could be changed in a later commit to make the openssh build consistently use _PATH_LOCALBASE. It is considered out-of-scope for this commit. Reviewed by: imp MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D26942 Notes: svn path=/head/; revision=367075
2020-08-17Don't explicitly specify c99 or gnu99 as the default is now gnu99.Xin LI
MFC after: 2 weeks Notes: svn path=/head/; revision=364292
2020-02-21fetch(3): plug some leaksKyle Evans
In the successful case, sockshost is not freed prior to return. The failure case can now be hit after fetch_reopen(), which was not true before. Thus, we need to make sure to clean up all of the conn resources which will also close sd. For all of the points prior to fetch_reopen(), we continue to just close sd. CID: 1419598, 1419616 Notes: svn path=/head/; revision=358227
2020-02-15fetch(3): don't leak sockshost on failureKyle Evans
fetch_socks5_getenv will allocate memory for the host (or set it to NULL) in all cases through the function; the caller is responsible for freeing it if we end up allocating. While I'm here, I've eliminated a label that just jumps to the next line... Notes: svn path=/head/; revision=357979
2020-02-15fetch(3): fix regression in IPv6:port spec from r357977Kyle Evans
In case the port was specified, we never actually populated *host. Do so now. Pointy hat: kevans Notes: svn path=/head/; revision=357978
2020-02-15fetch(3): move bits of fetch_socks5_getenv aroundKyle Evans
This commit separates out port parsing and validation from grabbing the host from the env var. The only related bit really is that we need to be more specific with the delimiter in the IPv6 case. Notes: svn path=/head/; revision=357977
2020-02-15fetch(3): Add SOCKS5 supportKyle Evans
This change adds SOCKS5 support to the library fetch(3) and updates the man page. Details: Within the fetch_connect() function, fetch(3) checks if the SOCKS5_PROXY environment variable is set. If so, it connects to this host rather than the end-host. It then initializes the SOCKS5 connection in accordance with RFC 1928 and returns the resulting conn_t (file descriptor) for usage by the regular FTP/HTTP handlers. Design Decision: This change defaults all DNS resolutions through the proxy by sending all IPs as hostnames. Going forward, another feature might be to create another environmental variable to toggle resolutions through the proxy or not.. One may set the SOCKS5_PROXY environment variable in any of the formats: SOCKS5_PROXY=proxy.example.com SOCKS5_PROXY=proxy.example.com:1080 SOCKS5_PROXY=192.0.2.0 SOCKS5_PROXY=198.51.100.0:1080 SOCKS5_PROXY=[2001:db8::1] SOCKS5_PROXY=[2001:db8::2]:1080 Then perform a request with fetch(1). (note by kevans) I've since been informed that Void Linux/xbps has a fork of libfetch that also implements SOCKS5. I may compare/contrast the two in the mid-to-near future. Submitted by: Farhan Khan <farhan farhan codes> Differential Revision: https://reviews.freebsd.org/D18908 Notes: svn path=/head/; revision=357968
2020-02-05libfetch: disallow invalid escape sequencesEd Maste
Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL. Suggested by: Matthew Dillon Reviewed by: Matthew Dillon MFC after: 1 week Notes: svn path=/head/; revision=357579
2020-01-28Fix urldecode buffer overrun.Gordon Tetlow
Reported by: Duncan Overbruck Security: CVE-2020-7450 Notes: svn path=/head/; revision=357212
2019-12-11Update Makefile.depend filesSimon J. Gerraty
Update a bunch of Makefile.depend files as a result of adding Makefile.depend.options files Reviewed by: bdrewery MFC after: 1 week Sponsored by: Juniper Networks Differential Revision: https://reviews.freebsd.org/D22494 Notes: svn path=/head/; revision=355617
2019-12-11Add Makefile.depend.optionsSimon J. Gerraty
Leaf directories that have dependencies impacted by options need a Makefile.depend.options file to avoid churn in Makefile.depend DIRDEPS for cases such as OPENSSL, TCP_WRAPPERS etc can be set in local.dirdeps-options.mk which can add to those set in Makefile.depend.options See share/mk/dirdeps-options.mk Reviewed by: bdrewery MFC after: 1 week Sponsored by: Juniper Networks Differential Revision: https://reviews.freebsd.org/D22469 Notes: svn path=/head/; revision=355616
2019-09-05pkgbase: Create a FreeBSD-utilities package and make it the default oneEmmanuel Vadot
The default package use to be FreeBSD-runtime but it should only contain binaries and libs enough to boot to single user and repair the system, it is also very handy to have a package that can be tranform to a small mfsroot. So create a new package named FreeBSD-utilities and make it the default one. Also move a few binaries and lib into this package when it make sense. Reviewed by: bapt, gjb Differential Revision: https://reviews.freebsd.org/D21506 Notes: svn path=/head/; revision=351858
2019-08-28Document fetchReqHTTP().Mark Johnston
Submitted by: Farhan Khan <khanzf@gmail.com> Reviewed by: 0mp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D18788 Notes: svn path=/head/; revision=351573
2019-05-03[libfetch] Fix compilation with WITHOUT_CRYPT.Adrian Chadd
Notes: svn path=/head/; revision=347050
2018-11-27When deciding whether to send the complete URL or just the document part,Dag-Erling Smørgrav
we were looking at the original URL rather than the one we were currently processing. This meant that if we were trying to retrieve an HTTP URL but were redirected to an HTTPS URL, and HTTPS proxying was enabled, we would send an invalid request and most likely get garbage back. MFC after: 3 days Notes: svn path=/head/; revision=341072
2018-11-27A few more cases where strcasecmp() is no longer required.Dag-Erling Smørgrav
MFC after: 1 week Notes: svn path=/head/; revision=341014
2018-11-27Improve URL parsing. In particular, convert scheme and host to lowercase.Dag-Erling Smørgrav
MFC after: 1 week Notes: svn path=/head/; revision=341013
2018-11-27Support proxying FTP over HTTPS, not just HTTP.Dag-Erling Smørgrav
There is probably a PR for this, but I can't find this, or remember who submitted it. The patch got lost in the noise of another that wasn't ready to commit. MFC after: 3 days Notes: svn path=/head/; revision=341011
2018-09-19Make libfetch buildable.Jung-uk Kim
Notes: svn path=/projects/openssl111/; revision=338779
2018-05-29Fix an inverted conditional in the netrc code, which would ignore theDag-Erling Smørgrav
value of $HOME and always use the home directory from the passwd database, unless $HOME was unset, in which case it would use (null). While there, clean up handling of netrcfd and add debugging aids. MFC after: 3 weeks Notes: svn path=/head/; revision=334326
2018-05-29Fix a few (but far from all) style issues.Dag-Erling Smørgrav
MFC after: 3 weeks Notes: svn path=/head/; revision=334319
2018-05-29Use __VA_ARGS__ to simplify the DEBUG macro.Dag-Erling Smørgrav
MFC after: 3 weeks Notes: svn path=/head/; revision=334317
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-20 19:10:51 +0000