# Make an instance of Redis Queue with connection to Redis serverīookInfoParserQueue = Queue( 'bookInfoParser',connection=redisClient) Here the method enqueue_call takes in a function that will be executed by a worker process. t(redisKey,pickle.dumps(bookInfo))Īfter validating the GoodReads URL, they are pushed to the RQ to parse the information.
# We use pickle.dumps method to convert the dictionary into a byte stream # which can be stored as a value in Redis against the key generated above # Here pickle module is used to serialize python objects. The function # returns a python dictionary.īookInfo = parse_book_link_for_meta_data(bookUrl) RedisKey = generate_redis_key_for_book(bookUrl) # get Redis key for given book URL # get book meta information from parsing function above. Generate_redis_key_for_book = lambda bookURL: 'GOODREADS_BOOKS_INFO:' + bookURLĭef parse_and_persist_book_info (bookUrl):
# This generates a unique Redis key against a book URL # Eg: For URL "", this lambda function # will return the key as "GOODREADS_BOOKS_INFO:" Here Docker # provides a link to our local redis server using 'redis' # Spawn a client connection to redis server. Generate_redis_key_for_book generates a unique key for a given book URL, since Redis is a key-value store that requires each key is unique. Now we can write a function called parse_and_persist_book_info to parse the above function and enqueue the value to the RQ for the worker processes. Return dict(title=title.strip() if title else '',author=author.strip() if author else '',rating=float(rating.strip() if rating else 0),description=description) Title = bsTree.find( "h1", attrs=).stripped_strings) HtmlString = requests.get(bookLink).content # Fetch HTML string of the book information pageīsTree = BeautifulSoup(htmlString, "html.parser") # Build a searchable tree using fetched HTML # Find the required book attributes in the tree # server.py def parse_book_link_for_meta_data (bookLink): BeautifulSoup allows us to search, manipulate and create structured markup languages to retrieve critical information such as the title, author, rating, and description. We'll use the requests library to make an HTTP request to GoodReads. Visit localhost:5000 in a browser to make sure the web service is working.
PYTHON SUBPROCESS RUN IN BACKGROUND CODE
The -build argument ensures the image is built using the latest code every time. This launches two containers, the web server application and Redis server interlinked using Docker networking. Start the Docker daemon and run docker-compose up -build to start the application. Setup the starter applicationįirst, clone the repository by running the following command: git clone
PYTHON SUBPROCESS RUN IN BACKGROUND INSTALL
you can head out here and install the relevant version. It requires Docker to be installed on your machine. You can use the starter repo here to follow along. They require a Redis server as a message broker to perform this operation. Redis Queue's allow you to enqueue function calls which can be executed in parallel by separate worker processes.
Since the function will take time to execute, we'll execute it asynchronously using Redis Queues (RQ) to prevent it from blocking the main thread. We'll write a function to crawl and parse the URL for the book's meta information. We will be writing a flask-based web application which retrieves Goodreads book information like title, author, rating, and description. Though our product doesn't have anything to do with task queues, we're a company that is trying to reinvent logging. This is a guest post brought to you by your friends Timber. These queues store messages or data incoming from producer microservices which can be processed by consumer microservices. Task queues are popular among microservice architectures because they enable each microservice to perform its dedicated task and they work as a medium for inter-microservice communication. This modular approach prevents the main process from becoming blocked. They run concurrently with the main process and execute the tasks present in the queue chronologically. Workers can be used to execute these tasks in the background. This requires an asynchronous strategy which uses queues to maintain a list of background tasks. It's often not feasible to execute a process when the request is received (especially when dealing with I/O) to prevent the thread from blocking incoming client requests.
0 kommentar(er)
