Support multiple domains

[acelaya]

Considerations:

• This feature must be backwards compatible. Any existing installation should keep working with currently configured domain.
• The domain that is currently provided to shlink will be considered the default one.
• It should be possible to create the same custom slug for different domains (which means the unique key currently set to the short_code column should be changed to use short_code+domain_id).
• Short codes will respond following these rules:
  • Request with unknown domain or default domain -> will redirect if the shortcode exists for default domain. 404 otherwise.
  • Request with know domain other than default one -> will redirect if shortcode is found for that domain or the default one. 404 otherwise.
• Domains should become a new entity.
• Domains will be created on the fly when new short URLs are created providing a specific domain (either via REST or CLI)
• Endpoints and commands to create short URLs should allow optionally providing the domain.
• When creating a short URL with findIfExists flag, it should try to match domain too, if provided.

Improvements for future versions:

• Allow to define different redirects for 404 errors on each one of the domains.
• Allow short codes to be restricted to one or multiple domains, failing to redirect instead of falling back to default one.
• Provide CLI commands and REST actions to create and list domains.
• Allow searching short URLs by domain name (including default domain, which value is not really stored)

Other affected projects:

• Shlink installer: Ask for domain name in a way that it's clear that more domnains can be added later shlink-installer#21
• Shlink web client: Allow domain to be defined when creating new short URLs shlink-web-client#144

[ghost]

Very happy to see this!

Additional feature requests:

1. Ability to specify a unique redirection URL for each domain if 404 link not found. If none specified for a custom domain, fall back on 'default' 404 URL.
2. Keep the behavior that any domain pointed to the Shlink service will grab the short code from the default domain. Meaning:

ShortDomainA.com/shortCode1 => Google. // Set up and working in Shlink
If you point ShortDomainB.com and point DNS to Shlink,
ShortDomainB.com/shortCode1 => Google // If the domain is not found, lookup shortCode against default domain anyway.

Does this make sense?

[acelaya]

@D-Visigo I think it mostly makes sense. I have updated the issue's description with some extra info.

2. Keep the behavior that any domain pointed to the Shlink service will grab the short code from the default domain.

This will be part of the first approach, as it's easier. In the future, I will probably allow restricting short codes on a per-domain basis.

1. Ability to specify a unique redirection URL for each domain if 404 link not found. If none specified for a custom domain, fall back on 'default' 404 URL.

This might be a little bit tricky, but makes sense. I will leave it out of v1.19, but include it soon after that.

I prefer to have a new version with less changes that will simplify testing the new feature and correcting anything that does not work as expected, while the feature is later improved.

[acelaya]

@D-Visigo Sadly, I didn't have much time to work on this the last weeks, but I have been thinking on it and how to better approach it.

The first thing that comes to my mind is what's really the benefit of adding multiple domains when you still want shlink to respond to any domain resolved via DNS, since that's how it currently behaves.

Maybe it is to make the API and CLI return the short URLs with the proper domain? Like this?:

Of course, I see the benefit of this feature when extended with the list of improvements for future versions, but since you specified this behavior, I want to make sure that I'm not missing anything.

[danielthedifficult]

Hi @acelaya ! Thanks for the update!

I'm sorry I wasn't more clear. The main reason for doing this is to be able to use multiple domains not just for branding purposes, but also to keep links short.

To be perfectly clear: I'm hoping to have the same 'shortcode' resolve to different destination URLs based on domain.

So that domainA.com/xyz can point to a different long URL than domainB.com/xyz

But, if a request comes in from a non-matching domain, it will use the 'default' domain for shortcode lookup.

I can explain the business case for this if needed, but I am essentially trying to increase the number of links I can manage in a single Shlink instance, and keep the links short, without having to delete older links too soon.

[acelaya]

Oh, I see. I definitely didn't understand that, but that's clearly a benefit I didn't think about.

However, I guess you are talking about custom slugs, and not automatically generated short codes, because those might or might not be the same. Is that correct?

I will redefine the minimum requirements for the first implementation and try to rethink the solution.

[danielthedifficult]

Yes, everything you said is correct :)

I am talking about both auto generated shortcodes and custom slugs.

For example, I might want domainA.com/unsubscribe to go to a different long URL than domainB.com/unsubscribe.

But I also want Shlink to keep a separate link 'namespace' for each domain - so auto-generated slugs in one domain don't require longer slugs in another domain :)

Let me see if I can better define the use cases:

Creation:
Case 1:
API request to shorten URL. No custom slug nor domain specified.
Expected outcome:
Shlink creates a random available slug in the default domain.

Case 2:
API request to shorten URL. Custom domain specified.
Expected Outcome:
Shlink creates a random available slug in the specified domain, ONLY valid for this specified domain (unless the custom domain specified is also the default domain).

Case 3:
API request to shorten URL. Custom domain and custom slug specified.
Expected Outcome:
Shlink creates a new shortlink using the custom slug in the specified domain, ONLY valid for this specified domain (unless the custom domain specified is also the default domain). Error if slug already exists.

Link Redirection
Case 1:
User arrives to shlink service with an unknown domain in the header.
Expected Outcome:
Shlink looks up the shortcode in the default domain table only, redirects if possible, 404 if the shortcode not found.

Case 2:
User arrives to shlink service with a known custom domain in the header.
Expected Outcome:
Shlink looks up the shortcode in the applicable custom domain table only, redirects if possible, 404 if the shortcode not found.

[acelaya]

Ok, I think I got it now. Just two observations:

The generation of short codes is not going to change, so there will never be two equal short codes attached to different domains. That would require several architectural changes and introduce technical debt to add very small benefit.

For custom slugs, it will work as you say.

The second thing is that I can see a third "link redirection" use case that I would like to clarify.

User arrives to shlink with a known custom domain. The shortcode/slug is not found for that domain but it is for the default one.

Should it fail with 404 or redirect to the link in the default domain?

[danielthedifficult]

Ouch! I was hoping that custom URL generation across domains would be easier.

Digging around the code, I seem to understand that you are using (UrlShortener.php line 87) getId() to get an "ID" of the generated shortcode - then you feed that to convertAutoincrementIdToShortCode to get a new 'slug'.

I was thinking the ID of each generated shortcode would be tracked/stored separately by domain. I admit I can't find where this float ID is coming from, but I guess I was hoping it was the same ID from the SQL table.

I guess that depends if you are planning on using one table per domain, or if you were going to put all shortURLs for all domains in the same table. I was expecting the former - but I'm interested to know what you're thinking.

EDIT: it looks like getId() might be a Zend thing? I don't have any experience with Zend so it's just a guess.

[danielthedifficult]

To answer the question in your last post, that is indeed a great question :)

Yes, the link should follow, but only for the default domain - not for the same valid slug on any other custom domain.

[acelaya]

Hey @danielthedifficult

You are right. Currently, the automatically generated short codes are generated based on the DB autoincrement ID and a shuffled charset which is currently based on the "base 62" charset 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ (an approach which I don't really like and will probably change in the near future. See #491). The getId() function is just returning that autoincrement database ID.

Since all short codes are stored in the same table, it would not be possible to "repeat" one. Since IDs are unique, short codes are globally unique too.

A small possibility to repeat one would happen if, as you suggest, different tables would be used for the short URLs of every existing domain, but this approach introduces a lot of problems:

• Having one domain entity (ShortUrl) which storage is spread across several tables will be hard to handle, and require custom code on top of the different third party libraries.
• Tables storing short URLs would need to be created/deleted dynamically, while new domains are added/removed. This would be quite hard to maintain and very error-prone.
• It would be a fairly overkill solution compared to solving it with simple foreign keys, which would be the normalized solution.
• When trying to find a short URL by its short code + domain, the table in which shlink should look for would need to be somehow guessed dynamically. In fact, several quieries would be needed in some cases.
• In the future, if some change is required in short URL tables, database migrations would have to dynamically find the tables to apply, which, again, would be a complex and error-prone operation.

If the reason to do this is to make sure short codes are kept as short as possible, you don't really need to worry about that.

The algorithm currently used by the UrlShortener::convertAutoincrementIdToShortCode method will return 4-character short codes for the first 39,000 short URLs created. After that 5-character short codes will be returned until you have more than 14,570,000 short URLs. I'm not even sure how many would be needed to jump to 7 characters, but you can make an idea.

I think it's not very likely that you reach those numbers, even with all short URLs saved in the same table 😅, so it's A LOT of technical debt for a very small benefit.

Yes, the link should follow, but only for the default domain - not for the same valid slug on any other custom domain.

Regarding to that ☝️, got it 😃.

[danielthedifficult]

Thank you very, very much for the detailed explanation. It is useful to understand, and I am learning a lot through this process 🙏🏻

Last question: How do you handle re-using shortcodes after they are deleted?

Meaning, let's say I create 1000 shortcodes, so I have ids 0-999.

I delete id's 100-200.

Will Shlink, either in SQL or in the application itself, insert to fill up id's 100-200 before continuing at #1000?

[acelaya]

Nop, no IDs or short codes will ever be reused, even if you delete existing short URLs (either by using shlink or manually deleting stuff from the database). This prevents unexpected side effects.

All supported databases have an autoincrement strategy which has memory on the latest value ever used, and it just "increments" the value on every new insert (that's why it's called autoincrement).

[acelaya]

BTW, progress on this feature is going well. I will most certenly have it done by the weekend.

I will probably publish a beta/RC before publishing the final v1.19, and it would be super helpful if you were able to help testing it. I'll let you know when it's available.

[acelaya]

@D-Visigo @danielthedifficult the basics of the feature are completed.

Tomorrow I will do a final review of the PR and probably merge it and publish a beta release.

[acelaya]

@D-Visigo @danielthedifficult I have just released v1.19.0-beta.1

• Release: https://github.com/shlinkio/shlink/releases/tag/v1.19.0-beta.1
• Docker image: https://hub.docker.com/r/shlinkio/shlink/tags?page=1&name=1.19

It comes with the multi domain support as described in this ticket. The web client changes have not been done yet, but you can already create short URLs for specific domains from the command line by passing --domain=example.com.

I will be testing the version, and if everything goes right, I'll puiblish the final version later today or tomorrow in the morning.

In the meantime I will be working on the web client changes, so that it is possible to provide the domain when creating short URLs.

Thanks for all the feedback and for using shlink 😃

[acelaya]

I have been implementing the feature in the web client, and while doing it, I have been testing the new version. It seems to work well, so I have already released the final stable version https://github.com/shlinkio/shlink/releases/tag/v1.19.0

The web client will be released this afternoon probably.

[danielthedifficult]

Hi Alejandro - Thank you so much. I'll hopefully be able to test and deploy this week, but I am working against several difficult deadlines ;-)

Because my main reason for wanting multi-domain support was to create new namespaces for links, there is now less pressure for this to move to production. I am exploring how to generate slugs on my 'client-side', and not use Shlink's auto-generation so I can recycle slugs across domains easily.

Thank you again!

[acelaya]

Sure! Feel free to test it whenever you have time.

[acelaya]

Hey @danielthedifficult!

I just wanted to let you know that the feature has been a little bit buggy since it was implemented. I didn't really anticipated it's real complexity and missed some considerations that didn't make it work as expected.

However, most of the issues have been solved in v2.0.4 and shlink-web-client v2.3.1, so you probably want to update to those if you still need to support multiple domains.

I have also added some documentation explaining how Shlink behaves when using multiple domains. You can find it here: Multiple domains.

Also, the last missing part from your original request, which is being able to define specific fall-back redirects for every domain, is already planned and will most probably land Shlink on v2.2.0.

I hope all of this helps 😃