Rendered at 19:12:01 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
EvanAnderson 1 days ago [-]
The relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit.
Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.
Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
stock_toaster 1 days ago [-]
Once dns-persist-01 becomes available/usable[1], it should make dns validation even easier.
Once it's supported I think my next iteration will be DNS persist + internal ip addresses on the public zone.
Thank you all for comments and feedback! It's cool to see real interest in this blog post
isomorphic 24 hours ago [-]
Thank you, I was unaware of that. It looks like it's already support in the acme.sh client, but there is a Let's Encrypt discussion saying it's still pending at LE:
I wonder if the interim version has been rolled out to some CAs.
theK 22 hours ago [-]
I never understood the issue with DNS-01. if you have a process that you trust to maintain a zone's TLS identity what is the big deal about letting it control a record in that zone?
nijave 21 hours ago [-]
We have a few subdomains for white labeling 3rd party SaaS where we do what is basically the AWS ACM equivalent and add a persistent record from a vendor.
With this setup, I don't have to grant 3rd parties DNS access.
I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values.
I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created.
I think the main takeaway is allowing pull instead of push model
Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case
bowersbros 10 hours ago [-]
You could already do that by setting a _acme-challenge alias CNAME record, so you can point it to a domain that they control. It's a bit finnicky and the new setup definitely looks better for whitelabelling, but there are current workarounds; although it does still involve client DNS access, you can put it onto a zone that has no risks
aeden 4 hours ago [-]
Our current level of granularity allows you to give read or write access to a specific zone, but it does not go down to the level of giving read or write access to a specific RRset type yet, if that's what you're looking for.
sigio 21 hours ago [-]
You can even put it in a seperate zone (which I do) by using a CNAME for _acme-challenge.domain.tld.
I have it to a seperate subdomain, which is served by a seperate desec.io account, which is only used for this specific subdomain.
mrl5 5 hours ago [-]
TIL, thanks! This makes dns challange automation more approachable for me. I always had an issue with API keys which scope can't be limited to TXT records. This seems to be a nice workaround
Another big problem with split-horizon is you can get clients that cache the dns result before connecting to the vpn, then after connecting, can't actually use the service, because it is using the public ip instead of the private one.
throw0101d 7 hours ago [-]
Just use IPv6: no "internal" (10/8) or "external" address, just an address. ;)
techcode 13 hours ago [-]
Yes - literally just pick the right Caddy (or similar) image with Let's Encrypt client included - and you can simply add a tag/label to your docker compose files for each of self hosted services to get a real SSL, that auto renewed ...etc.
With one of self hosted services being Adguard-Home can do both ADs blocking and internal DNS... The public DNS records for your "internal use only" domain remain empty.
s8kur 8 hours ago [-]
[dead]
selfmodruntime 1 days ago [-]
Also be careful when using split DNS and Tailscale, which increasingly won't work without MagicDNS enabled.
JoeBOFH 23 hours ago [-]
And you still have issues sometimes around dns not able to figure out which interface to use. We had issues around that before.
bruce511 1 days ago [-]
Came here to say the same. I use DNS-Challenge rather than HTTP-challenge, and that makes internal servers trivial.
You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.
To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.
throw0101d 7 hours ago [-]
> You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.
Library/CLI that speaks the API of several dozen DNS providers so you don't have to re-invent the wheel:
Interesting, this is the first time I've seen this project. One thing I saw that concerns me a bit is that they provide command-line options for passing your user credentials and multi-factor code. I'm not a fan of giving anything that level of access when a resource-specific access token will do, but that's me.
CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
bombcar 1 days ago [-]
Just LE a wild card cert and slap it everywhere.
EvanAnderson 1 days ago [-]
I get antsy about a private key being in lots of places. You've also got to worry about renewal so you might as well just have "consumer" of the wildcard just provision its own non-wildcard certificate.
sandermvanvliet 1 days ago [-]
I run a reverse proxy that handles the TLS termination for that reason.
Services themselves are constrained to the server, bound to listen on 127.0.0.1 only.
Key is only available to the reverse proxy.
jve 9 hours ago [-]
... Could you please provide a solution? What should I do in my homelab?
Saying something isn't bad without pointing to right direction makes my insides hurt a little bit.
EvanAnderson 7 hours ago [-]
I feel like I did. >smile<
> Use DNS validation to allow these internal services to pull ACME certs...
The major ACME clients support DNS validation. If your DNS host doesn't have an API that your ACME client supports either get a new ACME client or a new DNS host.
jve 6 hours ago [-]
That's for the cert part, yeah, got it.
I thought you have a solution to overcome not having to do split DNS where I define public IP in public DNS and internal IP within some local hosted DNS.
Like how to make so that when connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address?
EvanAnderson 5 hours ago [-]
Hairpin NAT (or "NAT hairpinning") is the term for "...connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address", and the Linux NAT implementation supports it.
For a homelab situation split-horizon DNS is just fine. You're going to have minimal duplication of records from the public DNS into the private DNS.
The canonical frustrating "bad split-horizon DNS" world I've seen, time and again, is a corporate network with a MSFT Active Directory named the same as the company's public-facing web property (e.g. "example.com" rather than "ad.example.com"). This creates the need to duplicate all the public-hosted resources into the internal "example.com" zone (and keep them updated when records inevitably change on the public Internet). It's make-work for no practical upside in that example.
jve 5 hours ago [-]
Okay, thanks. I thought split-DNS is a hack even at home.
Totally agree on the AD part - luckily we had an option to migrate to different domain and name our AD correctly :)
EvanAnderson 25 minutes ago [-]
Ha! AD domain renames in the mid-2000's made a decent chunk of money for me.
raffraffraff 1 days ago [-]
Hmm. I don't really care enough about leaking home network host names because they are all super generic names like 'router', 'laptop', 'tv', 'nas'. So I use my public zone on cloudflare. I just use internal ip addresses (eg: nas.example.com = 10.1.2.3) on the public zone and DNS01 challenge for let's encrypt. Anyone can resolve the ip for any of my hosts, but obviously you'd need to be on the wireguard vpn to hit them.
This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc
This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
graton 1 days ago [-]
I do the same thing. I'm not worried about them seeing my FQDNs.
I use the form of hostname.int.example.com for everything inside my home network. None of which is accessible to the outside world. I use LetsEncrypt with DNS validation to get the certificates.
tzs 19 hours ago [-]
If you are going to have all the home stuff on a subdomain (int.example.com) would it work to delegate int.example.com to a DNS server running at home what has internet access, and could handle the ACME DNS challenges for machines on int.example.com?
If it does then you don't have to mess with your public DNS whenever you want to add or renew certificates for home machines.
I'm using the free DNS my registrar provides, which doesn't provide API access unless you upgrade to their paid DNS service and so if I could use a local DNS server for the ACME challenges for the home network I could pick one that is friendly to automation.
So if you generally had "search example.com" in you resolv.conf, and were in the habit of having "web01.dev" in places, behaviour may have changed if you were suddenly on a machine that had the "search" line missing (or something else).
icedchai 6 hours ago [-]
What you describe is a user and resolver configuration problem. There are 100's of TLDs and there's always a chance they will conflict with subdomains, either now or in the future as new TLDs are created. I've been using both "int" and "dev" as subdomains since at least 2000 and never had an issue.
throw0101d 3 hours ago [-]
> What you describe is a user and resolver configuration problem.
That won't prevent me from getting a ticket saying "the network is down".
icedchai 3 hours ago [-]
Okay, I guess the only solution is to always use FQDNs.
0xbadcafebee 17 hours ago [-]
Which can still cause problems depending on your search domain setting and resolver client
icedchai 12 hours ago [-]
I do the same thing and have never had a problem. Maybe I’ve just been lucky for 25+ years. Some hosts have a search path of “int.example.com, example.com”. Others are just “example.com”
_bernd 11 hours ago [-]
Could you give an example? I'm curious, too.
throw0101d 7 hours ago [-]
If you generally had "search example.com" in you resolv.conf, and were in the habit of having "web01.dev" in places, behaviour may have changed if you were happen to be on a machine that had the "search" line missing (or something else).
graton 1 hours ago [-]
So? I don't see any issue.
I always use FQDNs for everything.
luckman212 1 days ago [-]
Fair, but what about names that are specific enough to give an attacker a clue to a potential attack surface, like "authelia.example.com" - now they know you've likely got an Authelia setup, and can start digging for exploitable CVEs etc. I'm in the process of removing all my individual certs and replacing with a wildcard cert served by Traefik. Is that a bad idea?
bbkane 1 days ago [-]
Can they dig for exploitable CVEs if they're not on the Wireguard network? It is a clue to your infrastructure, but I personally think the simplicity is worth it.
nijave 1 days ago [-]
My IaC is on public GitHub. They could do a network scan to find software then fingerprint to find version anyway.
Removing attack surface is better than trying to hide it.
akerl_ 20 hours ago [-]
How many people out there have attackers doing individualized research to identify services on their home LAN so they can chain a network attack with CVEs in their self-hosted service?
pixl97 2 hours ago [-]
Everyone, now that you can just toss the work at an LLM.
akerl_ 1 hours ago [-]
Where are you seeing this? LLMs make it easier to do bulk data analysis / scale attack patterns, but I've not seen anything to suggest they're incentivizing people to do OSINT against random individuals to fire off targeted attacks on home LANs.
The juice isn't really worth the squeeze for the token spend any more than it was worth the human energy.
icedchai 1 days ago [-]
Do the names resolve to publicly routeable IPs? If not, I wouldn't worry about it.
thayne 22 hours ago [-]
I don't think this is any less right than using split horizon. IMHO, there is no "right" way to do it. Every approach has downsides and tradeoffs.
icedchai 1 days ago [-]
This is similar to what I do, except I have my own authoritative DNS servers instead of Cloudflare.
I'd prefer this over split DNS, any day.
rao-v 24 hours ago [-]
At that point why not just use the .ts.net addresses Tailscale provides for free?
kennywinker 13 hours ago [-]
Because hostname.tail62bc83.ts.net is a mouthful.
rao-v 11 hours ago [-]
You can change it to something a tiny bit nicer a few times!
kennywinker 5 hours ago [-]
Oh! I did not know it was changeable. That’s useful!
boscillator 1 days ago [-]
The real answer here is that configuring HTTPS clients to trust a self-signed cert (or signed by an internal CA) shouldn't be as difficult as it is. I find it extremely annoying that every programming language has it's own idea of where certificates should live instead of just checking the os trust store.
greengreengrass 23 hours ago [-]
Possibly a separate concern, but I have some degree of confidence that the requirements and oversight of the CA/B forum (or whomever else determines which root certs go into bundles) are sufficiently strict, and issuers kept under sufficient scrutiny, that I tend to trust those more than I trust myself to secure my own root CA keys adequately. The ideal would be for people setting up their own PKI to ensure their root uses the Name Constraints extension, but the default “can sign anything for any host” I fear makes it easy for people to install their own self-pwn device, and probably left the private key lying around on a box exposed to the Internet.
* with some notable root certs that I have… questionable… trust and confidence are not simply controlled by certain state actors.
WhyNotHugo 19 hours ago [-]
Personally, I use a custom local CA with name constraints so that it can only sign domains for The .internal TLD. This is the most important bit: because if the cert is ever leaked, it cannot be used to MITM connections to other domains.
I have to secure the CA's key, but I also have to secure all the keys for the certificate it signs, both being a similar level of challenge.
For personal use, or for very small organisations, using a passphrase-protected Yubikey as a "cheap HSM" should suffice.
I am planning on fiddling with using the TPM of an old dell 5820 I've got floating around my house as a budget HSM.
It looks possible, on paper, if you don't poke at it too often.
thayne 16 hours ago [-]
I wish that in addition to the CA setting Name Constraints (and client software validating that), that end users could add additional constraints when adding a new trusted CA, so that even if the CA cert doesn't have Name Constraints, you can restrict it to a specific domain.
afarah1 1 days ago [-]
Even if all applications look at the OS trust store, in my experience there's always a gap distributing the CA to every consumer, leading to time spent on debugging from time to time... Maybe that's not the case in perfectly homogeneous or sufficiently small environments where every team uses the same infra / stack.
WhyNotHugo 19 hours ago [-]
Yeah, adding CAs to the store sucks. On Linux it needs to be in /etc/ssl/certs (this varies slighly per distribution), which is only writable by root. A single user can't trivially trust a CA, and a great deal of applications/libraries don't support overriding the store's path.
thayne 16 hours ago [-]
It's worse than that. OpenSSL uses /etc/ssl/certs (or sometimes a different directory, depending on the distro) by default. But Java uses its own store that uses an incompatible java-specific format, and Chromium derivatives and Firefox have their own trust stores. So to add a CA to a system it is often necessary to add it in multiple places.
skywhopper 1 days ago [-]
Is there a commonly used language other than Java that doesn’t just defer to the OS trusted CAs by default?
michaelt 20 hours ago [-]
The behaviour of curl depends on how your version was built.
Python? The widely used 'requests' relies on 'certifi' and skips the OS store - but 'pip' on the other hand does use the OS store.
A tool that might use java, like a database or IDE, means it might have its own store.
Node? Make sure you set NODE_USE_SYSTEM_CA=1
Firefox and Chrome AFAIK both have their own stores.
Building a Docker container? That's intentionally isolated from the host, of course your container won't inherit the OS trusted CAs. Running a VM locally? Same.
Installed something using snap? The container-like isolation means it won't pick up the OS trusted CAs.
And of course you need the certs set up right on your cloud servers, your CI servers, the dozen different smartphones the mobile team uses for testing.
brewmarche 20 hours ago [-]
Python for example, although nowadays you can simply install pip_system_certs to change that behaviour.
A lot of non-language tools bring their own certificate bundle as well, like uv, git, curl and Firefox. (I think they might all be the same Mozilla bundle even).
However, it seems like the situation here has improved slightly: git can read Windows certificates now once a flag has been set, Firefox has a flag for Windows and macOS, and it supports p11-kit. curl can be built with Windows/macOS support and respects OpenSSL environment variables. uv can be configured to use system-certs.
And obviously any VM or container has to be set up separately. That often includes stuff like pipelines in forges. As mentioned in the sibling comment, at least here it's definitely by design.
jeroenhd 22 hours ago [-]
Node and (probably by extension) Electron infuriate me with their custom requirements.
Several Rust libraries also tend to default to a predefined set of certificates (which makes sense for libraries supposed to run on bare metal as there are no system certificates there, but that's not really a problem on most Linux installs). I make it a point to always use the native OS roots in the code I write, but unfortunately that's not universal.
Having to bind-mount certificates inside of docker containers is also always an annoyance I forget about until I see the first TLS errors in the logs, but that's by design and probably a good thing.
spacebanana7 1 days ago [-]
It's hard to get right when OSs, programming languages, browsers and sometimes other applications have their own opinions about trust stores. I understand why our IT department want corporate devices to use internal CA certs on paper but it just breaks stuff in the real world.
patrakov 1 days ago [-]
My preferred procedure is to use DNS-01 validation and have no publicly accessible "A" or "AAAA" record for internal services.
Or even a more extreme example: https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
otabdeveloper4 1 days ago [-]
Sounds bonkers. Why not make an overlay LAN and host your own DNS server in 10.0.0.0/8?
patrakov 1 days ago [-]
I do have a DNS server in my LAN, with some records served to internal clients only. But the _acme-challenge record needs to be public for the DNS-01 validation to succeed.
The point was that you can obtain a certificate for a domain name without creating any records other than the _acme-challenge TXT record. I.e., that the domain might be completely empty all the time except for this record.
samgranieri 1 days ago [-]
Split horizon DNS is not something I'm willing to do.
I'll just rock out with .internal or .home.arpa, have step-ca and bind communicate to each other, either in step-issuer in kube or maybe even rfc2136 if i feel like a bit of the dns-01 strategy is in order. I slap the internal ca root certs everywhere, and keep my home infra out of the crt.sh logs.
I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
wrxd 1 days ago [-]
I use the acme dns-1 challenge on my public domain. That gives you certificates you can use as you see fit, without needing to expose anything else to the public internet.
I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
throw0101d 1 days ago [-]
> I use the acme dns-1 challenge on my public domain.
See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain:
So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net").
There are DNS servers written strictly focused on this use case:
- Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network.
- Don't use the HTTP-01 challenge. Use DNS-01.
- Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.
Walf 17 hours ago [-]
>Don't use split DNS
So what's your solution when you have a wholly private service that will never have a public v4 address, nor a publicly routable v6? How do clients get the address for a nice domain name without the addresses in public DNS?
I use acme.sh with DNS validation, and use common domains that have both public and private services on subdomains. I use split horizon so private.domain.example resolves only on LAN and VPN, and public.domain.example resolves everywhere, but the address changes depending on the network one is connected to.
master_crab 7 hours ago [-]
Wait…what? You have a fully internal service and you dont have an internal DNS server? I guess those exist in theory, but not in practice. :)
If you have a heavy enough tech stack to run fully internal services, than you can also run an internal DNS service (even pihole is enough) and load internal only entries there.
Or add everything to your hosts file if you have a central config service.
xurukefi 13 hours ago [-]
> How do clients get the address for a nice domain name without the addresses in public DNS?
They don’t. You put the addresses in public DNS.
Walf 12 hours ago [-]
Yeah, nah. Won't be doing that.
vaughnegut 8 hours ago [-]
Why not? It's easy to set up and manage and it just points at an internal IP address
jeroenhd 22 hours ago [-]
Using wildcard certs and/or a central reverse proxy defeats the purpose of internal TLS.
Split-horizon DNS for a publicly usable domain is almost always a bad idea, but running your own ACME server is pretty easy (maybe 10 lines of Caddy config) and using an internal domain (an actual one, not a randomly picked TLD you don't think exists yet) solves the problem pretty easily. You'll want a safe backup for your root certificate private key, of course, but that's pretty much all you need to really worry about.
Plasmoid 21 hours ago [-]
Let's encrypt is great but if you operate any kind of scale you can quickly hit their rate limit. AWS only recently started allowing you to pull the certificate out of ACM.
preisschild 23 hours ago [-]
> Use a central reverse proxy/load balancer for termination.
If you do that anyways, you could also use something like oidc authN/authZ on the reverse proxy level and just expose it to the internet.
You dont even need to self host the oidc idp, you can use Google/Github or even something like ATProto
thomashabets2 1 days ago [-]
Personally, I hate split horizon DNS. I prefer the "BeyondCorp" model. I MUCH prefer putting an mTLS cert in my trusted devices over relying on VPNs in same devices. I've yet to see a "clever" DNS setup not cause annoyances.
Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.
Zero trust or bust. I actually cringed when I read "done right" applied to split-horizon DNS.
lee_ars 3 hours ago [-]
OP, why mess with acme.sh's cron schedule with your own cron job that calls its cron job? You might instead consider running acme.sh again with the "--install-cert" flag, coupled with its "--reloadcmd" flag, in order to automatically have it do the cert+key copying and nginx reloading for you when it renews:
Then you can ditch your custom cron and let acme handle everything on its own, as intended.
pizzalife 1 days ago [-]
I don't agree that tunneling everything through some external facing proxy is "TLS certificates for internal services done right".
nijave 1 days ago [-]
Arguably it's 1/2. You can put public certs on proxy then give proxy private CA to backend services. Then you don't need public certs for all the private stuff nor need to trust the private CA on all your devices.
xorcist 1 days ago [-]
This is crazy. If you have a home network with a few internal services, or some sort of network where you don't control the endpoints, just use DNS validation. That's why it exists.
But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
maqnius 1 days ago [-]
Can you elaborate why one shouldn't use DNS validation for hosts you control in general?
xorcist 1 days ago [-]
That's not it. If you have an internal network, where every host is provisioned by you, you already control identity.
In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.
llama052 1 days ago [-]
There's certainly something to be said for ease of use and not having to ensure you push trusted certs to every device that touches your internal network.
Unless you enjoy that sort of thing.
master_crab 10 hours ago [-]
Split brain =/= done right.
This introduces all sorts of aberrant behavior. Most notably that clients will get different responses depending on their query and route that you now have to test for. It also is a “silent” deviation for audiences that may not be aware of.
I’m not sure what problem that is trying to be solved here? That the OP wanted internal users seeing a different site than external users? Or using a different route in? In both those cases the correct user behavior is to use a different DNS record.
jmbwell 1 days ago [-]
I don’t know much in this space, but I find myself wishing there was a dead simple self hosted CA solution and also that trust on first use (à la ssh) was A Thing for self-managed root certs in client implementations. TOFU is such an elegant, good-enough solution for these use cases. Fixed deployment is always still an option, but in this day and age it feels so much like we are unnecessarily still dealing with solved problems
nijave 1 days ago [-]
On k8s, there's cert-manager but also you need k8s...
Most browsers support trust on first use for leaf certs
jmbwell 1 days ago [-]
I guess I mean treat it as a clear first class feature. Right now most browsers treat it as an arcane error. I’m thinking more “This is the first time you’re connecting to this site. Do you trust it?”
And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today
nijave 20 hours ago [-]
Yeah, that's fair. I think they're optimized for non technical users without a decent escape hatch.
notTooFarGone 1 days ago [-]
Firefox now started that you can't even go on the page on some occasions.
Using a browser in an air gapped environment is so much more pain than it should be.
bigfatkitten 22 hours ago [-]
That support is limited. Browsers refuse to support passkeys when you TOFU untrusted certs, and for good reason.
zufallsheld 1 days ago [-]
Well, if your cert-manager distributes its own CA, you'd still need the clients to trust the CA, even in k8s.
nijave 21 hours ago [-]
True but you can have cert-manager issue public certs then create service accounts for off cluster things to be able to pull the cert from the Secret so k8s+cert-manager acts as a local broker that handles renewal.
You can also invert and have k8s cronjobs provision the generated certs into other infra
With this setup, you don't have to worry about the RHEL certbot snap updating to a broken version which gets blocked by SELinux...
sandeepkd 1 days ago [-]
I was under impression that I understand networking and DNS resolution. It was really hard to follow, the OP did worked hard, just not sure what exact problem was being solved with the proposed solution that isnt already been solved.
raquuk 1 days ago [-]
I am looking forward to finally using DNS-PERSIST-01 for validation. No more dynamic DNS updates, DNS credentials or forwarding necessary.
And then the issue is protecting the private key of the issuer and monitoring certificates (it's a good idea to do that anyway).
mnahkies 13 hours ago [-]
Personally I think wildcard certs are more secure in many instances - if you're just running one load balancer/reverse proxy then all the private keys are going to the same place anyway. You get the upside of not leaking your subdomains to transparency logs, and having fewer certs to renew.
It also forces you to use DNS challenges, which means you don't require the publicly route able address.
Personally what I do is:
- Predominantly use wildcards
- Run a tiny coredns instance with my A records for a internal subdomain of my normal domain
- Configure tailscale to use the coredns resolver
- Run two haproxy instances, one for internal services, one for public facing. The public facing one can't route directly to the internal services.
When even the obscure DNS provider I'm using is supported for DNS challenges, I really don't see much upside to using HTTP challenges anymore
aliasxneo 1 days ago [-]
I wonder if the author realizes that getting public certificates results in them being recorded in CT logs.
Thom2000 8 hours ago [-]
Yep. Especially with non-wildcard certs this leaks all service names (privacy concern).
cobertos 1 days ago [-]
Why not just map the domain to an internal IP and call it a day? Then the only way it can be accessed is through a VPN. Then use a wildcard so none of leaks into cert transparency logs
throw0101d 1 days ago [-]
> Then use a wildcard so none of leaks into cert transparency logs
You now also have to build infrastructure to distribute the wildcard from (presumably) central place where you generate it to all the different places where it is desired.
And hope the wildcard's private key does not leak from one of myriad of places it now lives.
cobertos 1 days ago [-]
I have a few Traefik instances that request wildcards independently of each other. Each with the same config, per server.
Leaking is an issue but we're talking about internal services too.
AlexanderYamanu 1 days ago [-]
well, I configure my services te request their own wildcard certs from a caching proxy acme to letsencrypt. Easy peasy.
sigio 21 hours ago [-]
My setup is having a wildcard DNS record and a wildcard certificate for my 'home' domain. It has a fixed IP from my ISP, so you always end up on haproxy, which then forwards to individual ports/ip's in the internal network.
I can do filtering based on source-ip from there, so traffic from myself/internal will be allowed, and outside traffic (not from some allowlisted ip's) will get blocked.
ACME validation is done via DNS, so nothing needs to be accessable for that to issue certificates. Internally I will usually also use the public IP for services, so no need for a split-dns.
ramblurr 1 days ago [-]
> TLS certificates for internal services* done right
* "internal services" = on a single server that is publicly routable
28304283409234 23 hours ago [-]
When using letsencrypt for internal services it becomes very hard to distinguish badplayer.com LE certificate from the good players. You only ensure encryption. Not identity. Do not use letsencrypt internally for things that matter.
spydum 1 days ago [-]
this is all fine and good, if you are okay broadcasting your internal hostnames.
I suppose it's a trade off some might make.
CartwheelLinux 1 days ago [-]
There's one way around that which is requesting a wildcard cert, but then that has its own rammifications
gh02t 1 days ago [-]
There's really two ways, the other is to manage your own CA. But it seems like every browser/piece of software/etc out there is hell bent on making that as difficult as possible. It'd also be nice if it was easier to scope a certificate authority to a specific domain, but support for that is pretty patchy which is functionally the same as no support at all. And that's not to mention software that ignores the system certificate store. Or how tedious and nonstandardized it can be to get a trusted certificate store in a Docker container in cases where you have services that need to trust each other. Or how annoying it is to install your own trusted CA on devices (though, step-cli does help a lot at least on normal computers... phones however...). On and on and on, the barriers to what should be the obvious solution are extremely high.
mrl5 1 days ago [-]
I've documented how to securely set up TLS certificates for internal services without creating TLS issues for http clients downstream. All thanks to split-horizon DNS, WAF and ACME protocol. All for free!
rootnod3 1 days ago [-]
Uhm....source?
cwillu 1 days ago [-]
He's the submitter
woranl 17 hours ago [-]
I use client certificate authentication (or mTLS) to access internal services. It requires the client device to present a digital certificate during the TLS handshake, so I don’t need to white list any IP addresses.
jabart 1 days ago [-]
A Github Action running acme.sh that pushes certs to S3 solves the split dns issue for hosts, which can cause all sorts of weirdness after a while. You can then grab a cert on a schedule and even make them wildcard if you want. Then you will get NXDOMAIN if you are not on the VPN so ideally no public traffic.
llama052 1 days ago [-]
DNS-01 validation is way less work than that in my experience.
jabart 23 hours ago [-]
Thats what this Github Action uses is DNS validation through acme.sh and updating a Route53 domain.
gmuslera 1 days ago [-]
"Right" without use case can be wrong. And by use case I include scale. For a small team, few machines, some in-place infrastructure may worth it. Smaller than that may be overkill, bigger than that may not be enough, or end being cumbersome, insecure or not work for everyone.
Eduard 1 days ago [-]
to help with passive reconnaissance, here are tuxnet.dev's SSL certificates and associated subdomains:
I use a registered domain with DNS validation and then CNAMEs that I resolve locally. Basically:
1. Register a domain ("server.com") and put it on some public DNS that can do DNS validation with acme.sh.
2. Use DNS validation to get a certificate on your domain from Let's Encrypt. You can just grab a wildcard one ("*.server.com").
3. CNAME all of your services on a public DNS to an internal address ("email.server.com" → "server.internal", "plex.server.com" → "server.internal").
4. Resolve your internal address on a local DNS server with an A record ("server.internal" → 192.168.0.123). This can often just be done on your router.
Since you use DNS validation, you just API keys for your public DNS service that acme.sh can use. No need to have any VPN network interfaces for getting your certificate. Your wildcard certificate also doesn't leak any details about your services.
iambvk 17 hours ago [-]
I like this approach. Thank you.
Would it work if a user's device that is already connected to the VPN, but has custom DNS override to say 8.8.8.8 ? How can I allow my users to be able to use 8.8.8.8 DNS override and still work seamlessly?
Hamuko 14 hours ago [-]
I think you'd at least have to have the fallback DNS set to your local DNS resolver in order to get the server.internal resolved to a A record.
denkmoon 20 hours ago [-]
Bogus nonsense. Sign your internal certs with an internal CA, make your clients trust the CA. Use it properly.
0010010111 22 hours ago [-]
Feasible in local (7) btrfs.
AtNightWeCode 1 days ago [-]
Don't do this. Public certs are for public services.
cpach 1 days ago [-]
Why does it matter?
guptadagger 21 hours ago [-]
im not understanding why you would need a waf at all
Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.
Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
[1]: https://letsencrypt.org/2026/02/18/dns-persist-01
Once it's supported I think my next iteration will be DNS persist + internal ip addresses on the public zone.
Thank you all for comments and feedback! It's cool to see real interest in this blog post
https://community.letsencrypt.org/t/dns-persist-01-deploymen...
I wonder if the interim version has been rolled out to some CAs.
With this setup, I don't have to grant 3rd parties DNS access.
I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values.
I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created.
I think the main takeaway is allowing pull instead of push model
Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case
... and it is even already documented at https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
With one of self hosted services being Adguard-Home can do both ADs blocking and internal DNS... The public DNS records for your "internal use only" domain remain empty.
You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.
To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.
Library/CLI that speaks the API of several dozen DNS providers so you don't have to re-invent the wheel:
* https://github.com/dns-lexicon/dns-lexicon
https://github.com/acme-dns/acme-dns
CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
Services themselves are constrained to the server, bound to listen on 127.0.0.1 only.
Key is only available to the reverse proxy.
Saying something isn't bad without pointing to right direction makes my insides hurt a little bit.
> Use DNS validation to allow these internal services to pull ACME certs...
The major ACME clients support DNS validation. If your DNS host doesn't have an API that your ACME client supports either get a new ACME client or a new DNS host.
I thought you have a solution to overcome not having to do split DNS where I define public IP in public DNS and internal IP within some local hosted DNS.
Like how to make so that when connecting from inside local network the router recognize that by connecting to public IP, he has to route it back onto some local IP address?
For a homelab situation split-horizon DNS is just fine. You're going to have minimal duplication of records from the public DNS into the private DNS.
The canonical frustrating "bad split-horizon DNS" world I've seen, time and again, is a corporate network with a MSFT Active Directory named the same as the company's public-facing web property (e.g. "example.com" rather than "ad.example.com"). This creates the need to duplicate all the public-hosted resources into the internal "example.com" zone (and keep them updated when records inevitably change on the public Internet). It's make-work for no practical upside in that example.
Totally agree on the AD part - luckily we had an option to migrate to different domain and name our AD correctly :)
This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc
This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
I use the form of hostname.int.example.com for everything inside my home network. None of which is accessible to the outside world. I use LetsEncrypt with DNS validation to get the certificates.
If it does then you don't have to mess with your public DNS whenever you want to add or renew certificates for home machines.
I'm using the free DNS my registrar provides, which doesn't provide API access unless you upgrade to their paid DNS service and so if I could use a local DNS server for the ACME challenges for the home network I could pick one that is friendly to automation.
Note that int is a valid TLD:
* https://en.wikipedia.org/wiki/.int
* https://datatracker.ietf.org/doc/html/rfc1591
Lots of folks were using "dev" as a sub-domain which was fine until ICANN decide to give Google a TLD:
* https://en.wikipedia.org/wiki/.dev
So if you generally had "search example.com" in you resolv.conf, and were in the habit of having "web01.dev" in places, behaviour may have changed if you were suddenly on a machine that had the "search" line missing (or something else).
That won't prevent me from getting a ticket saying "the network is down".
I always use FQDNs for everything.
Removing attack surface is better than trying to hide it.
The juice isn't really worth the squeeze for the token spend any more than it was worth the human energy.
I'd prefer this over split DNS, any day.
* with some notable root certs that I have… questionable… trust and confidence are not simply controlled by certain state actors.
I have to secure the CA's key, but I also have to secure all the keys for the certificate it signs, both being a similar level of challenge.
For personal use, or for very small organisations, using a passphrase-protected Yubikey as a "cheap HSM" should suffice.
* https://www.yubico.com/products/hardware-security-module/
See also perhaps less expensive option:
* https://shop.nitrokey.com/shop/nkhs2-nitrokey-hsm-2-7
It looks possible, on paper, if you don't poke at it too often.
Python? The widely used 'requests' relies on 'certifi' and skips the OS store - but 'pip' on the other hand does use the OS store.
A tool that might use java, like a database or IDE, means it might have its own store.
Node? Make sure you set NODE_USE_SYSTEM_CA=1
Firefox and Chrome AFAIK both have their own stores.
Building a Docker container? That's intentionally isolated from the host, of course your container won't inherit the OS trusted CAs. Running a VM locally? Same.
Installed something using snap? The container-like isolation means it won't pick up the OS trusted CAs.
And of course you need the certs set up right on your cloud servers, your CI servers, the dozen different smartphones the mobile team uses for testing.
A lot of non-language tools bring their own certificate bundle as well, like uv, git, curl and Firefox. (I think they might all be the same Mozilla bundle even).
However, it seems like the situation here has improved slightly: git can read Windows certificates now once a flag has been set, Firefox has a flag for Windows and macOS, and it supports p11-kit. curl can be built with Windows/macOS support and respects OpenSSL environment variables. uv can be configured to use system-certs.
And obviously any VM or container has to be set up separately. That often includes stuff like pipelines in forges. As mentioned in the sibling comment, at least here it's definitely by design.
Several Rust libraries also tend to default to a predefined set of certificates (which makes sense for libraries supposed to run on bare metal as there are no system certificates there, but that's not really a problem on most Linux installs). I make it a point to always use the native OS roots in the code I write, but unfortunately that's not universal.
Having to bind-mount certificates inside of docker containers is also always an annoyance I forget about until I see the first TLS errors in the logs, but that's by design and probably a good thing.
Or even a more extreme example: https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
The point was that you can obtain a certificate for a domain name without creating any records other than the _acme-challenge TXT record. I.e., that the domain might be completely empty all the time except for this record.
I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain:
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net").
There are DNS servers written strictly focused on this use case:
* https://github.com/acme-dns/acme-dns
Also code that handles a bunch of DNS provider APIs so you don't have to roll your own for ACME client hooks:
* https://github.com/dns-lexicon/dns-lexicon
https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
- Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network.
- Don't use the HTTP-01 challenge. Use DNS-01.
- Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.
So what's your solution when you have a wholly private service that will never have a public v4 address, nor a publicly routable v6? How do clients get the address for a nice domain name without the addresses in public DNS?
I use acme.sh with DNS validation, and use common domains that have both public and private services on subdomains. I use split horizon so private.domain.example resolves only on LAN and VPN, and public.domain.example resolves everywhere, but the address changes depending on the network one is connected to.
If you have a heavy enough tech stack to run fully internal services, than you can also run an internal DNS service (even pihole is enough) and load internal only entries there.
Or add everything to your hosts file if you have a central config service.
They don’t. You put the addresses in public DNS.
Split-horizon DNS for a publicly usable domain is almost always a bad idea, but running your own ACME server is pretty easy (maybe 10 lines of Caddy config) and using an internal domain (an actual one, not a randomly picked TLD you don't think exists yet) solves the problem pretty easily. You'll want a safe backup for your root certificate private key, of course, but that's pretty much all you need to really worry about.
If you do that anyways, you could also use something like oidc authN/authZ on the reverse proxy level and just expose it to the internet.
You dont even need to self host the oidc idp, you can use Google/Github or even something like ATProto
Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.
I bet you can do it with HA-Proxy, but I use https://github.com/ThomasHabets/sni-router
acme.sh --install-cert -d grafana.tuxnet.dev --key-file /etc/ssl/private/grafana.tuxnet.dev.key --fullchain-file /etc/ssl/certs/grafana.tuxnet.dev.crt --reloadcmd "systemctl reload nginx"
Then you can ditch your custom cron and let acme handle everything on its own, as intended.
But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.
Unless you enjoy that sort of thing.
This introduces all sorts of aberrant behavior. Most notably that clients will get different responses depending on their query and route that you now have to test for. It also is a “silent” deviation for audiences that may not be aware of.
I’m not sure what problem that is trying to be solved here? That the OP wanted internal users seeing a different site than external users? Or using a different route in? In both those cases the correct user behavior is to use a different DNS record.
Most browsers support trust on first use for leaf certs
And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today
Using a browser in an air gapped environment is so much more pain than it should be.
You can also invert and have k8s cronjobs provision the generated certs into other infra
With this setup, you don't have to worry about the RHEL certbot snap updating to a broken version which gets blocked by SELinux...
And then the issue is protecting the private key of the issuer and monitoring certificates (it's a good idea to do that anyway).
It also forces you to use DNS challenges, which means you don't require the publicly route able address.
Personally what I do is: - Predominantly use wildcards
- Run a tiny coredns instance with my A records for a internal subdomain of my normal domain
- Configure tailscale to use the coredns resolver
- Run two haproxy instances, one for internal services, one for public facing. The public facing one can't route directly to the internal services.
When even the obscure DNS provider I'm using is supported for DNS challenges, I really don't see much upside to using HTTP challenges anymore
You now also have to build infrastructure to distribute the wildcard from (presumably) central place where you generate it to all the different places where it is desired.
And hope the wildcard's private key does not leak from one of myriad of places it now lives.
Leaking is an issue but we're talking about internal services too.
* "internal services" = on a single server that is publicly routable
https://www.certkit.io/tools/ct-logs/?query=tuxnet.dev
Would it work if a user's device that is already connected to the VPN, but has custom DNS override to say 8.8.8.8 ? How can I allow my users to be able to use 8.8.8.8 DNS override and still work seamlessly?