Free Resources
Increasing Conversion With Web Performance
Recommended Articles
An Introduction to Laravel Queues and Temporary URLs
Laravel is a mature, robust, and powerful web framework that makes developing PHP applications a breeze. In particular, I want to demonstrate how to create a website that can be used to convert videos online using queue jobs for processing and temporary URLs for downloading the converted files. This article is aimed at those who aren’t very familiar with Laravel yet. Prerequisites There are many ways to set up Laravel, and which is the best method may depend on your operating system or preference. I have found Laravel Herd to be very easy to use if you’re using Windows or macOS. Herd is a Laravel development environment that has everything you need with minimal configuration required. Command-line tools are installed and added to your path, and background services are configured automatically. If you’re developing on Linux then Herd is not an option. However, Laravel Sail works for all major operating systems and uses a Docker based environment instead. You can find a full list of supported installation methods in the Laravel documentation. To keep things simpl,e this article assumes the use of Herd, though this won’t make a difference when it comes to implementation. You will also need a text editor or IDE that has good PHP support. PhpStorm is a great editor that works great with Laravel, but you can also use VSCode with the Phpactor language server, and I’ve found Phpactor to work quite well. Project Setup With a development environment setup, you can create a new Laravel project using composer, which is the most popular package manager for PHP. Herd installs composer for you. composer installs dependencies and lets you run scripts. Let’s create a Laravel project using it: ` Once that is done you can navigate into the project directory and start the server with artisan: ` Awesome! You can now navigate to http://localhost:8000/ and see the Laravel starter application’s welcome page. Artisan is the command-line interface for Laravel. It comes with other utilities as well such as a database migration tool, scripts for generating classes, and other useful things. Uploading Videos Using Livewire Livewire is a library that allows you to add dynamic functionality to your Laravel application without having to add a frontend framework. For this guide we’ll be using Livewire to upload files to our server and update the status of the video conversion without requiring any page reloads. Livewire can be installed with composer like so. ` With it installed we need to make a Livewire component now. This component will act as the controller of our video upload page. ` With that done you should see two new files were created according to the output of the command, one being a PHP file and the other being a Blade file. Laravel has its own HTML template syntax for views that allow you to make your pages render dynamically. For this demo we’ll make the video conversion page render at the root of the site. You can do this by going to routes/web.php and editing the root route definition to point to our new component. ` However, if we visit our website now it will return an error. This is due to the app template being missing, which is the view that encapsulates all page components and contains elements such as the document head, header, footer, etc. Create a file at resources/views/components/layouts/app.blade.php and put the following contents inside. This will give you a basic layout that we can render our page component inside of. ` The {{ $slot }} string in the main tag is a Blade echo statement. That is where our Livewire component will be injected when loading it. Now, let’s edit the Livewire component’s template so it has something meaningful in it that will allow us to verify that it renders correctly. Edit resources/views/livewire/video-uploader.blade.php and put in the following: ` With that done you can go to the root of the site and see this hello message rendered inside of a box. Seeing that means everything is working as it should. We may as well delete the welcome template since we’re not using it anymore. This file is located at resources/views/welcome.blade.php. Now, let’s go ahead and add uploading functionality. For now we’ll just upload the file into storage and do nothing with it. Go ahead and edit app/Livewire/VideoUploader.php with the following: ` This will only allow uploading files with video file MIME types. The $video class variable can be wired inside of the component’s blade template using a form. Create a form in resources/views/livewire/video-uploader.blade.php like so: ` You will note a wire:submit attribute attached to the form. This will prevent the form submission from reloading the page and will result in Livewire calling the component’s save method using the video as a parameter. The $video property is wired with wire:model="video". Now you can upload videos, and they will be stored into persistent storage in the storage/app/private directory. Awesome! Increase the Filesize Limit If you tried to upload a larger video you may have gotten an error. This is because the default upload size limit enforced by Livewire and PHP is very small. We can adjust these to accommodate our use-case. Let’s start with adjusting the Livewire limit. To do that, we need to generate a configuration file for Livewire. ` All values in the generated file are the defaults we have been using already. Now edit config/livewire.php and make sure the temporary_file_upload looks like this: ` The rules key allows us to change the maximum file size, which in this case is 100 megabytes. This alone isn’t good enough though as the PHP runtime also has a limit of its own. We can configure this by editing the php.ini file. Since this article assumes the use of Herd, I will show how that is done with it. Go to Herd > Settings > PHP > Max File Upload Size > and set it to 100. Once done you need to stop all Herd services in order for the changes to take effect. Also make sure to close any background PHP processes with task manager in-case any are lingering, as this happened with me. Once you’ve confirmed everything is shut off, turn on all the services again. If you’re not using Herd, you can add the following keys to your php.ini file to get the same effect: ` Creating a Background Job Now, let’s get to the more interesting part that is creating a background job to run on an asynchronous queue. First off, we need a library that will allow us to convert videos. We’ll be using php-ffmpeg. It should be noted that FFmpeg needs to be installed and accessible in the system path. There are instructions on their website that tell you how to install it for all major platforms. On macOS this is automatic if you install it with homebrew. On Windows you can use winget. On macOS and Linux you can confirm that ffmpeg is in your path like so: ` If a file path to ffmpeg is returned then it’s installed correctly. Now with FFmpeg installed you can install the PHP library adapter with composer like so: ` Now that we have everything we need to convert videos, let’s make a job class that will use it: ` Edit app/Jobs/ProcessVideo.php and add the following: ` To create a job we need to make a class that implements the ShouldQueue interface and uses the Queueable trait. The handle method is called when the job is executed. Converting videos with php-ffmpeg is done by passing in an input video path and calling the save method on the returned object. In this case we’re going to convert videos to the WebM container format. Additional options can be specified here as well, but for this example we’ll keep things simple. One important thing to note with this implementation is that the converted video is moved to a file path known by the livewire component. Later and to keep things simple we’re going to modify the component to check this file path until the file appears, and while for demo purposes this is fine, in an app deployed at a larger scale with multiple instances it is not. In that scenario it would be better to write to a cache like Redis instead with a URL to the file (if uploaded to something like S3) that can be checked instead. Now let’s use this job! Edit app/Livewire/VideoUploader.php and let’s add some new properties and expand on our save method. ` How this works is we tell the job where it can find the video and tell it where it should output the converted video when it’s done. We have to make the output filename be the same as the original with just the extension changed, so we use pathinfo to extract that for us. The ProcessVideo::dispatch method is fire and forget. We aren’t given a handle of any kind to be able to check the status of a job out of the box. For this example we’ll be waiting for the video to appear at the output location. To process jobs on the queue you need to start a queue worker as jobs are not processed in the same process as the server that we are currently running. You can start the queue with artisan: ` Now the queue is running and ready to process jobs! Technically you can upload videos for conversion right now and have them be processed by the job, but you won’t be able to download the file in the browser yet. Generating a Temporary URL and Sending it with Livewire To download the file we need to generate a temporary URL. Traditionally this feature has only been available for S3, but as of Laravel v11.24.0 this is also usable with the local filesystem, which is really useful for development. Let’s add a place to render the download link and the status of the job. Edit resources/views/livewire/video-uploader.blade.php and add a new section under the form: ` Note the wire:poll attribute. This will cause the Blade echo statements inside of the div to refresh occasionally and will re-render if any of them changed. By default, it will re-render every 2.5 seconds. Let’s edit app/Livewire/VideoUploader.php to check the status of the conversion, and generate a download URL. ` Every time the page polls we check if the video has appeared at the output path. Once it’s there we generate the link, store it to state, and pass it to the view. Temporary URLs are customizable as well. You can change the expiration time to any duration you want, and if you’re using S3, you can also pass S3 request parameters using the optional 3rd argument. Now you should be able to upload videos and download them with a link when they’re done processing! Limitations Although this setup works fine in a development environment with a small application, there are some changes you might need to make if you plan on scaling beyond that. If your application is being served by multiple nodes then you will need to use a remote storage driver such as the S3 driver, which works with any S3 compatible file storage service. The same Laravel API calls are used regardless of the driver you use. You would only have to update the driver passed into the Storage facade methods from local to s3, or whichever driver you choose. You also wouldn’t be able to rely on the same local filesystem being shared between your job workers and your app server either and would have to use a storage driver or database to pass files between them. This demo uses the database driver for simplicity’s sake, but it's also worth noting that by default, queues and jobs use the database driver, but SQS, Redis, and Beanstalkd can also be used. Consider using these other drives instead of depending on how much traffic you need to process. Conclusion In this article, we explored how to utilize queues and temporary URLs to implement a video conversion site. Laravel queues allow for efficient processing of long-running tasks like video conversion in a way that won’t bog down your backend servers that are processing web requests. While this setup works fine for development, some changes would need to be made for scaling this such as using remote storage drivers for passing data between the web server and queue workers. By effectively leveraging Laravel’s features, developers can create robust and scalable applications with relative ease....
Feb 4, 2025
9 mins
The Importance of a Scientific Mindset in Software Engineering: Part 1 (Source Evaluation & Literature Review)
The Importance of a Scientific Mindset in Software Engineering: Part 1 (Source Evaluation & Literature Review) Today, I will write about something very dear to me - science. But not about science as a field of study but rather as a way of thinking. It's easy nowadays to get lost in the sea of information, fall for marketing hype, or even be trolled by a hallucinating LLM. A scientific mindset can be a powerful tool for navigating the complex modern world and the world of software engineering in particular. Not only is it a powerful tool, but I'll argue that it's a must nowadays if you want to make informed decisions, solve problems effectively, and become a better engineer. As software engineers, we are constantly confronted with an overwhelming array of frameworks, technologies, and infrastructure choices. Sometimes, it feels like there's a new tool or platform every day, each accompanied by its own wave of hype and marketing. It's easy to feel lost in the myriad of information or even suffer from FOMO and insecurity about not jumping on the latest bandwagon. But it's not only about the abundance of information and making technological decisions. As engineers, we often write documentation, blog posts, talks, or even books. We need to be able to communicate our ideas clearly and effectively. Furthermore, we have to master the art of debugging code, which is essentially a scientific process where we form hypotheses, test them, and iterate until we find the root cause of the problem. Therefore, here's my hot take: engineering is a science; hence, to deserve an engineer title, one needs to think like a scientist. So, let's _review_ (pun intended) what it means to think like a scientist in the context of software engineering. Systematic Review In science, systematic review is not only an essential means to understand a topic and map the current state of knowledge in the field, but it also has a well-defined methodology. You can't just google whatever supports your hypothesis and call it a day. You must define your research question, choose the databases you will search, set your inclusion and exclusion criteria, systematically search for relevant studies, evaluate their quality, and synthesize the results. Most importantly, you must be transparent about and describe your methodology in detail. The general process of systematic review can be summarized in the following steps: 1. Define your research question(s) 2. Choose databases and other sources to search 3. Define keywords and search terms 4. Define inclusion and exclusion criteria a. Define practical criteria such as publication date, language, etc. b. Define methodological criteria such as study design, sample size, etc. 5. Search for relevant studies 6. Evaluate the quality of the studies 7. Synthesize the results Source: Conducting Research Literature Reviews: From the Internet to Paper by Dr. Fink I'm pretty sure you can see where I'm going with this. There are many use cases in software engineering where a process similar to systematic review can be applied. Whether you're evaluating a new technology, choosing a tech stack for a new project, or researching for a blog post or a conference talk, it's important to be systematic in your approach, transparent about your methodology, and honest about the limitations of your research. Of course, when choosing a tech stack to learn or researching for a blog post, you don't have to be as rigorous as in a scientific study. But a few of these steps will always be worth following. Let's focus on those and see how we can apply them in the context of software engineering. Defining Your Research Question(s) Before you start researching, it's important to define your research questions. What are you trying to find out? What problem are you trying to solve? What are the goals of your research? These questions will help you stay focused and avoid focusing on irrelevant information. > A practical example: If you're evaluating, say, whether to use bundler _A_ or bundler _B_ without a clear research question, you might end up focusing on marketing claims about how bundler _A_ is faster than bundler _B_ or how bundler _B_ is more popular than bundler _A_, even though such aspects may have minimal impact on your project. With a clear research question, you can focus on what really matters for your project, like how well each bundler integrates with your existing tools, how well they handle your specific use case, or how well they are maintained. A research question is not a hypothesis - you don't have to have a clear idea of the answer. It's more about defining the scope of your research and setting clear goals. It can be as simple and general as "What are the pros and cons of using React vs. Angular for a particular project?" but also more specific and focused, like "What are the legal implications of using open-source library _X_ for purpose _Y_ in project _Z_?". You can have multiple research questions, but keeping them focused and relevant to your goals is essential. In my personal opinion, part of the scientific mindset is automatically having at least a vague idea of a research question in your head whenever you're facing a problem or a decision, and that alone can make you a more confident and effective engineer. Choosing Databases and Other Sources to Search In engineering, some information (especially when researching rare bugs) can be scarce, and you have to search wherever and take what you can get. Hence, this step is arguably much easier in science, where you can include well-established databases and publications in your search. Information in science is simply more standardized and easier to find. There are, however, still some decisions to be made about where to search. Do you want to include community websites like StackOverflow or Reddit? Do you want to include marketing materials from the companies behind the technologies you're evaluating? These can all be valid sources of information, but they have their limitations and biases, and it's important to be aware of them. Or do you want to ask a LLM? I hadn't included LLMs in the list of valid sources of information on purpose as they are not literature databases in the traditional sense, and I wouldn't consider them a search source for literature research. And for a very good reason - they are essentially a black box, and therefore, you cannot reliably describe a reproducible methodology of your search. That doesn't mean you shouldn't ask an LLM for inspiration or a TL;DR, but you should always verify the information you get from them and be aware of their limitations. Defining Keywords and Search Terms This section will be short, as most of you are familiar with the concept of keywords and search terms and how to use search engines. However, I still wanted to highlight the importance of knowing how to search effectively for a software engineer. It's not just about typing in a few keywords and hoping for the best. It's about learning how to use advanced search operators, filter out irrelevant results, and find the information you need quickly and efficiently. If you're not familiar with advanced search operators, I highly recommend you take some time to learn them, for example, at FreeCodeCamp. Please note, however, that the article is specific to Google and different search engines may have different operators and syntax. This is especially true for scientific databases, which often have their own search syntax and operators. So, if you're doing more formal research, familiarize yourself with the database's search syntax. The underlying principles, however, are pretty much the same everywhere; just the syntax and UI might differ. With a solid search strategy in place, the next critical step is to assess the quality of the information we find. Methodological Criteria and Evaluation of Sources This is where things get interesting. In science, evaluating the quality of the studies is a crucial step in the systematic review process. You can't just take the results of a study at face value - you need to critically evaluate its design, the sample size, the methodology, and the conclusions - and you need to be aware of the limitations of the study and the potential biases that may have influenced the results. In science, there is a pretty straightforward yet helpful categorization of sources that my students surprisingly needed help understanding because no one ever explained it to them. So let me lay out and explain the three categories to you now: 1. Primary sources Primary sources represent original research. You can find them in studies, conference papers, etc. In science, this is what you generally want to cite in your own research. However, remember that only some of what you find in an original research paper is a primary source. Only the parts that present the original research are primary sources. For example, the introduction can contain citations to other studies, which are not primary, but secondary sources. While primary sources can sometimes be perceived as hard to read and understand, in many cases, they can actually be easier to reach and understand as the methods and results are usually presented in a condensed form in the abstract, and often you can only skim the introduction and discussion to get a good idea of the study. In software engineering, primary sources can sometimes be papers, but more often, they are original documentation, case studies, or even blog posts that present original research or data. For example, if you're evaluating a new technology, the official documentation, case studies, and blog posts from its developers can be considered primary sources. 2. Secondary sources Secondary sources are typically reviews, meta-analyses, and other sources that summarize, analyze, or reference the primary sources. A good way to identify a source as secondary is to look for citations to other studies. If a claim has a citation, it's likely a secondary source. On the other hand, something is likely wrong if it doesn't have a citation and doesn't seem to present original research. Secondary sources can be very useful for getting an overview of a topic, understanding the current state of knowledge, and finding relevant primary sources. Meta-analyses, in particular, can provide a beneficial point of view on a subject by combining the results of multiple studies and looking for patterns and trends. The downside of secondary sources is that they can introduce information noise, as they are basically introducing another layer of interpretation and analysis. So, it's always a good idea to go back to the primary sources and verify the information you get from secondary sources. Secondary sources in software engineering include blog posts, talks, or articles that summarize, analyze, or reference primary sources. For example, if you're researching a new technology, a blog post that compares different technologies based on their documentation and/or studies made by their authors can be considered a secondary source. 3. Tertiary sources Tertiary sources represent a further level of abstraction. They are typically textbooks, encyclopedias, and other sources that summarize, analyze, or reference secondary sources. They are useful for getting a broad overview of a topic, understanding the basic concepts, and finding relevant secondary sources. One example I see as a tertiary source is Wikipedia, and while you shouldn't ever cite Wikipedia in academic research, it can be a good starting point for getting an overview of a topic and finding relevant primary and secondary sources as you can easily click through the references. > Note: It's fine to reference Wikipedia in a blog post or a talk to give your audience a convenient explanation of a term or concept. I'm even doing it in this post. However, you should always verify that the article is up to date and that the information is correct. The distinction between primary, secondary, and tertiary sources in software engineering is not as clear-cut as in science, but the general idea still applies. When researching a topic, knowing the different types of sources and their limitations is essential. Primary sources are generally the most reliable and should be your go-to when seeking evidence to support your claims. Secondary sources can help get an overview of a topic, but they should be used cautiously, as they can introduce bias and noise. Tertiary sources are good for getting a broad overview of a topic but should not be used as evidence in academic research. Evaluating Sources Now that we have the categories laid out let's talk about evaluating the quality of the sources because, realistically, not all sources are created equal. In science, we have some well-established criteria for evaluating the quality of a source. Some focus on the general credibility of the source, like the reputation of the journal or the author. In contrast, others focus on the quality of the study itself, like the study design, the sample size, and the methodology. First, we usually look at the number of citations and the impact factor of the journal in which the study was published. These numbers can give us an idea of how well the scientific community received the study and how much other researchers have cited it. In software engineering, we don't have the concept of impact factor when it comes to researching a concept or a technology, but we can still look at how many people are sharing the particular piece of information and how well the professional community receives it and how reputable the person sharing the information is. Second, we look at the study design and the methodology. Does the study have a clear research question? Is the study design appropriate for the research question? Are the methods well-described and reproducible? Are the results presented clearly and honestly? Do the data support the conclusions? Arguably, in software engineering, the honest and clear presentation of the method and results can be even more important than in science, given the amounts of money circulating in the industry and the potential for conflicts of interest. Therefore, it's important to understand where the data is coming from, how it was collected, and how it was analyzed. If a company (or their DevRel person) is presenting data that show their product is the best (fastest, most secure...), it's important to be aware of the potential biases and conflicts of interest that may have influenced the results. The ways in which the results can be skewed may include: - Missing, incomplete, or inappropriate methodology. Often, the methodology is not described in enough detail to be reproducible, or the whole experiment is designed in a way that doesn't actually answer the research question properly. For example, the methodology can omit important details, such as the environment in which the experiment was conducted or even the way the data was collected (e.g., to hide selection bias). - Selection bias can be a common issue in software engineering experiments. For example, if someone is comparing two technologies, they might choose a dataset that they expect to perform better with one of the technologies or a metric that they expect to show a difference. Selection bias can lead to skewed results that don't reflect the technologies' real-world performance. - Publication bias is a common issue in science, where studies that show a positive result are more likely to be published than studies that show a negative outcome. In software engineering, this can manifest as a bias towards publishing success stories and case studies, while ignoring failures and negative results. - Confirmation bias is a problem in science and software engineering alike. It's the tendency to look for evidence that confirms your hypothesis and ignore evidence that contradicts it. Confirmation bias can lead to cherry-picking data, misinterpreting results, and drawing incorrect conclusions. - Conflict of interest. While less common in academic research, conflicts of interest can be a big issue in industry research. If a company is funding a study that shows its product in a positive light, it's important to be aware of the potential biases that may have influenced the results. Another thing we look at is the conclusions. Do the data support the conclusions? Are they reasonable and justified? Are they overstated or exaggerated? Are the limitations of the study acknowledged? Are the implications of the study discussed? It all goes back to honesty and transparency, which is crucial for evaluating the quality of the source. Last but not least, we should look at the citations and references included in the source. In the same way we apply the systematic review process to our research, we should also apply it to the sources we use. I would argue that this is even more important in software engineering, where the information is often less standardized, and you come across many unsupported claims. If a source doesn't provide citations or references to back up their claims, it's a red flag that the information may not be reliable. This brings us to something called anecdotal evidence. Anecdotal evidence is a personal story or experience used to support a claim. While anecdotal evidence can be compelling and persuasive, it is generally considered a weak form of evidence, as it is based on personal experience rather than empirical data. So when someone tells you that X is better than Y because they tried it and it worked for them, or that Z is true because they heard it from someone, take it with a massive grain of salt and look for more reliable sources of information. That, of course, doesn't mean you should ask for a source under every post on social media, but it's important to recognize what's a personal opinion and what's a claim based on evidence. Synthesizing the Results Once you have gathered all the relevant information, it's time to synthesize the results. This is where you combine all the evidence you have collected, analyze it, and draw conclusions. In science, this is often done as part of a meta-analysis, where the results of multiple studies are combined and analyzed to look for patterns and trends using statistical methods. A meta-analysis is a powerful tool for synthesizing the results of multiple studies and drawing more robust conclusions than can be drawn from any single study. You might not be doing a formal meta-analysis in software engineering, but you can still apply the same principles to your research. Look for common themes and trends in the information you have gathered, compare and contrast different sources, and draw conclusions based on the evidence. Conclusion Adopting a scientific way of thinking isn't just a nice-to-have in software engineering - it's essential to make informed decisions, solve problems effectively, and navigate the vast amount of information around you with confidence. Applying systematic review principles to your research allows you to gather reliable information, evaluate it critically, and draw sound conclusions based on evidence. Let's summarize what such a systematic research approach can look like: - Define Clear Research Questions: - Start every project or decision-making process by clearly stating what you aim to achieve or understand. - Example: "What factors should influence our choice between Cloud Service A and Cloud Service B for our application's specific needs?" - Critically Evaluate Sources: - Identify the type of sources (primary, secondary, tertiary) and assess their credibility. - Be wary of biases and seek out multiple perspectives for a well-rounded understanding. - Be Aware of Biases: - Recognize common biases that can cloud judgment, such as confirmation or selection bias. - Actively counteract these biases by seeking disconfirming evidence and questioning assumptions. - Systematically Synthesize Information: - Organize your findings and analyze them methodically. - Use tools and frameworks to compare options based on defined criteria relevant to your project's goals. I encourage you to embrace this scientific approach in your daily work. The next time you're facing a critical decision - be it selecting a technology stack, debugging complex code, or planning a project - apply these principles: - Start with a Question: Clearly define what you need to find out. - Gather and Evaluate Information: Seek out reliable sources and scrutinize them. - Analyze Systematically: Organize your findings and look for patterns or insights. - Make Informed Decisions: Choose the path supported by evidence and sound reasoning. By doing so, you will enhance your problem-solving skills and contribute to a culture of thoughtful, evidence-based practice in the software engineering community. The best part is that once you start applying a critical and systematic approach to your sources of information, it becomes second nature. You'll automatically start asking questions like, "Where did this information come from?" "Is it reliable?" and "Can I reproduce the results?" Doing so will make you much less susceptible to hype, marketing, and new shiny things, ultimately making you happier and more confident. In the next part of this series, we'll look at applying the scientific mindset to debugging and using hypothesis testing and experimentation principles to solve problems more effectively....
Jan 10, 2025
14 mins