This tutorial will show how to build a URL shortener similar to TinyURL, covering both the design and implementation aspects.
Technologies
Golang
Gin Framework
GORM
Redis
PostgreSQL
High-level design
API Endpoints
API endpoints enable communication between clients and servers. We will design the APIs following the REST architecture. If you’re not familiar with RESTful APIs, you can refer to external resources, such as restfulapi. A URL shortener mainly requires two API endpoints.
URL shortening: To generate a new short URL, the client makes a POST request that includes a single parameter: the original long URL.
URL redirecting: To redirect a short URL to its associated long URL, the client issues a GET request.
URL shortening
To support the URL shortening use case, we have to find a hash function that maps a long URL to the short URL. We are assuming the hash table stores <shortURL, longURL> pairs.
URL redirecting
The API supports two status codes for redirects: 301 (permanent redirect) and 302 (temporary redirect):
301 redirect: A 301 redirect indicates that the requested URL has been “permanently” moved to the long URL. Because of the permanent nature of the redirect, the browser caches the response, and future requests for the same URL bypass the URL shortening service, directing traffic straight to the long URL’s server.
302 redirect: A 302 redirect indicates that the URL has been “temporarily” moved to the long URL. As a result, future requests for the same URL will continue to go through the URL shortening service first before being redirected to the long URL’s server.
To reduce the server load, 301 redirect is a good choice when only the first request of the same URL is sent to URL shortening service.
Low-level design
Hash algorithm
As with the normal hash algorithm, we will have some collisions. Each time of this, it is expensive to query the database to check if a shortURL exists for every query. A common approach used for this is base62 conversion. Base62 conversion is used as there are 62 possible characters [0-9, a-z, A-Z] for hashValue. The example below to illustrates how the conversion works:
I decided to use the Twitter snowflake approach to generate a unique ID as a input of hash function. I’ll explain the reasons for this choice in a future blog post to avoid making this one too lengthy.
URL shortening flow
As one of the core pieces of the system, I build the diagram below to demonstrate the flow.
Verify if the longURL is present in the cache and return the corresponding shortURL. If it’s not in the cache, check the database, store it in the cache, and return it to the user. If it’s still not found, treat the longURL as new and proceed to step 2
Generate a new unique ID using the Twitter Snowflake algorithm and use it as input for the Base62 function. Then proceed to step 3
Store the <shortURL, longURL\> mapping in the database and cache, then provide the new shortURL to the user
URL redirecting flow
A user accesses shortURL
The load balancer forwards the request to web servers
If a shortURL already in the cache, return the longURL directly
If a shortURL is not in the cache. fetch the longURL from the database
Return 301 status code with location longURL to the user
Development
Step 1: Set up the project
Open your terminal and create a new directory for the project, then init a Go module to manage dependencies
Step 2: Define the URL struct
We create a URL struct to manage the mapping between shortURL and longURL
Step 3: Implement URL Shortening
We created a GenerateShortURL function that processes POST requests, generates a unique short URL, and returns a JSON response to the user.
Step 4: Implement URL redirecting
We created a RedirectShortURL function that handles GET requests, takes a short URL as a parameter and redirects to the corresponding long URL if it exists.
Notes:
The caching method is managed within the service layer for each function call to it.
This code above is just for illustration, the real code may be differ than that. You can get full source code here
Conclusion
That is it for this article. I hope you found this article useful, if you need any help please let me know by directly inbox.
Let's connect on Twitter and LinkedIn.
👋 Thanks for reading, See you next time