Unveiling the Secrets of Event-Driven Architecture and Technologies
Welcome to the Joshua Matos show your place for diving deep into the vast field of software engineering I'm Joshua Matos your full-stack software engineer and I'm here to make sure you become the most awesome software engineer today's episode we'll be exploring event driven architecture event dispatching spring cloud dataflow and advanced message queuing protocol we'll touch on some of the cloud services you can use too so what is event driven architecture why should we care about it and most excitingly what can we achieve with it suppose you didn't know how to leverage these powerful Tools in that case you can transform your organization by enabling them to implement microservices event driven architecture and most importantly allow your teams to make real-time data-driven decisions knowing about event-driven architecture and how to leverage message broker systems will make you an invaluable asset as a software engineer event driven architecture Eda is a software design Paradigm centered on events driving communication for instance an item purchased at a counter triggers an event to update the inventory is design really emphasizes the production detection and consumption of events events are triggered communication is made decouple Services collaborate and the systems rely on eventual consistency we'll talk a little bit more about eventual consistency later Eda decouples services and enables asynchronous communication which makes your environment scalable this decoupling mechanism alleviates bottlenecks your application doesn't have to wait for a message to be received applications continue with their workload while sending information elsewhere to be processed imagine if the customer had to wait for the inventory item to be updated before being able to make the purchase applications become more resilient to failure as Services can operate independently if a service encounters an issue the rest of your system will continue to run this spot tolerant nature ensures the systems remain functional delivering consistent and uninterrupted Services event-driven applications core strength lies in its ability to build systems that respond to events real time by proactively reacting to events applications can provide instantaneous updates real-time notifications and dynamic responses enhancing user experience and driving data decision making it's worth noting that spring boot uses concepts of event dispatching to enable the publication of events to other services this powerful pattern can be implemented with or without a message broker system streaming is another technique within Eda enables the processing of events as they occur such as financial data or sensor data streams can be used for filtering data data enrichment and other Technologies like machine learning in an event-driven architecture separate microservices may be responsible for their part of the domain it could be an inventory equipment or a user domain and an Eda these systems would either publish or consume events the messages can be sent to All Systems only one or several and this type of architecture communication is asynchronous events May not always arrive simultaneously your customer may purchase an item that sends an event that updates the inventory and cart system these messages will not necessarily arrive at the same time which is otherwise known as eventual consistency Eda is powerful but not ubiquitous so you may have to move your organization or software development team toward an event driven architecture one issue you may encounter when introducing an event-driven architecture into your company is that the current code base is monolithic meaning the applications are tightly coupled the front end and back end services are tied closely often in the same repository monoliths are usually deployed as a single individual unit shifting to an event-driven architecture creates a more resilient and scalable infrastructure the first step often involves moving to microservices simply separating the back and front ends can most easily accomplish this doing something as simple as this can help shift your organization mindset on designing and coding software another approach is to create a new service independent of the application does the application require a new inventory service create a separate backend only that responds to inventories as time passes you can introduce new technologies that facilitate your microservices and enables event-driven architecture let's talk about spring boot's event dispatching and how to use it to improve your software development events are a tool that allow different parts of your application to communicate with each other and help ensure your application remains decoupled often when one part of your application needs to collaborate with another part you'll inject one class into another and then call one of its methods or an API is introduced into an application used throughout the code base essentially coupling your application tightly and spring boot you can Implement event dispatching to help overcome this Pitfall by extending the application abstract class or using another pre-built type such as the application event publisher class that lets you publish events right into spring context events can fire off to another service that can exist inside your code base or with minimal configuration externally these are many use cases for event dispatching it's vital that you know what they are the more you know about it the better you can use event dispatching to log events in your application this can help with debugging and monitoring depending on your environment you can come up with exciting use cases such as sending logs to a database for storage or a security service event dispatching can notify users on other systems when certain events occur this can be used for sending email notifications when a new user registers or SMS notifications when a product is out of stock event dispatching can coordinate the behavior of different parts of your application for example you could use event dispatching for adding a student to a university and a roster for a university class and two separate databases in your architecture you'll likely start with this approach as it's the simplest implement you'll rely on events to execute other parts of your code and only verify the event was triggered in a unit or integration test event dispatching is a powerful technique that can make your application more modular reusable and easier to test if you still need to start using event dispatching I encourage you to try it you may be surprised how much it can improve your code the caveat being is that it can also complicate your code if it isn't clear that other events are being triggered now when you start leaning towards an event driven architecture there are some questions you should ask yourself how resilient are my events can I scale my systems depending on the use case how do we make events more resilient a great way to make a more resilient event driven architecture is to offload your events to another service asynchronously this is ampq or Advanced messaging queuing protocol at the heart of asynchronous communication protocols ampq is a protocol that standardizes how messages should be transported between applications like the shipping lanes of our ocean mpq specifies how messages are formatted addressed transported and confirmed ensuring reliable and predictable point-to-point or point-to-multipoint Communications the protocol is designed around these Core Concepts exchanges cues bindings and topics exchanges are entry points for messages there are akin to Ports of countries or messages or ships first arrive from a publisher cues are buffers that store messages if exchanges are ports cues are the harbors where our ships are stored until they can be processed messages are processed in order first in first out just like the grocery check outline bindings are a type of role that exchanges use to Route messages into queues think of them as InterContinental routes that defined where each ship should go the routing key determines which column the message should go to whether it's one queue or many ampq has different types of exchanges namely four direct topic fan out and headers typically you hear about topics a form of routing where messages are routed based on a match between a message routing key and the pattern used to bind a queue to an exchange the routing key is a short string the message producer generates specifying the topic to which the message belongs it can contain wow cards for example the routing key system.status would match the pattern system.astric wildcard which would route the message to any queue bound to the exchange with the same pattern direct exchanges also allow the message to be routed on the specific routing key it's a simple unicast routing if you have a queue whose binding is the same that's where it will go fan out exchanges where the routing key is ignored the message is sent out to all cues bound to The Exchange there is a scenario where you need to send a message to multiple cues this is the one for you header exchanges rely on header values instead of routing keys of course for all these messages being routed you'll have Publishers and consumers of those messages Publishers create messages and send them to exchanges exchanges use bindings are rules to decide how to route messages to cues that end up at a consumer a monitoring service can subscribe to a topic that starts with a key and a wild card and consume relevant messages such as system.status one of the
major strengths of ampq is its reliability the protocol guarantees messages delivery through acknowledgment if a message isn't acknowledged the protocol knows the message wasn't correctly delivered and tries again this ensures that no message gets lost and to stay analogous is to say it ensures our ships aren't lost and the vast ocean void enhancing the communication and system reliability in an event-driven architecture the reliability of messages is essential ensuring an event was received as an integral part of designing an ADA now let's talk about RAB and mq a popular message broker that implements the ampq protocol rapid mq at its core is an open source message broker software that implements the advanced message queuing protocol it's designed to enable Communications between different parts of a system if you wanted to ensure your spring blue application events were robust and communicate between dispersed and separate systems you'd use a message broker like rabbitmq if you have a server located somewhere else or want to share with a system in the cloud such as an S3 bucket or a database in a kubernetes pod picture rabbitmq as an Intergalactic post office your applications or planets if we stick to our Cosmic metaphor sends messages to the message broker rabbitmq ensuring they are delivered to the correct recipients no matter where their system is located in the solar system one of the greatest strengths of rabbitmq is its support for various messaging patterns rabbitmq has work or task queues if you have a resource intensive task you don't want to wait to complete you can send it as a message to a work queue where multiple workers will execute it and it does all the functions you would expect from ampq message broker such as publish subscribing routing and using topics rabbitmq is resilient to failure it guarantees that a message will be delivered even if the receiver is not ready when it's sent or if some other parts of the system have crashed rapidmq can be integrated with spring boot enabling your applications to communicate using events resiliently spring boot provides abstractions and Integrations to work with rampa mq namely the spring and PQ project it provides templates for sending and receiving messages containers and makes it easier to implement in your project rabbitmq added stream support in version 3.11 they offer large fan outs allowing multiple applications to read the same message it also has time travel so that streams can replay data streams have timestamps where a part of the stream can be picked and the data replayed streams are persistent and in rabbitmq replicate a non-destructive append Only log these features for rapidmq are a game changer and helped it to compete with Apache Kafka normally in a rabbitmq scenario messages are deleted from the queue whenever a message is consumed however streams allow a consumer to go to any point in a log and read from there rabbitmq was designed to have any empty State however now with stream support you can have large backlogs that store data there's so much more to rabbitmq it's a great option for anyone looking to implement a message broker into their architecture now let's take a deeper look into Apache Kafka Apache Kafka is a distributed event streaming platform born out of LinkedIn to handle their growing data and activity and simple terms Kafka allows us to send messages from one application to another from one system to another in real time using a publish And subscribe model as well much like a courier service it ensures our messages or events reach their intended recipient but Kafka is not just a messaging system it's a streaming platform that can store process and analyze streams of Records in real time it can handle trillions of daily events making it a favor among Tech giants like Twitter Netflix and Uber Apache Kafka is particularly valuable and modern applications for several reasons its ability to handle real-time data makes it ideal for event-driven architecture where the goal is to detect and react to changes in the state of data as soon as they occur let's say you have a retail application every time a customer places an order this generates an event Kafka and can ingest this event and notify various parts of the application perhaps the event system needs to update its stock the Billing System needs to issue an invoice and the customer needs a confirmation email this can be handled with Kafka allowing for real-time efficient processing of course the billing and inventory and customer systems would listen for the event when each received the order event they would act whether it was updating their stock issuing an invoice or sending an email confirmation the retail application is effectively decoupled from the rest of the system we could scale and evolve each system independently of each other you can then link this information to a visualization dashboard to view real-time analytics and provide your organization with business insights events would be stored in a queue your dashboard of choice or multiple dashboards would subscribe to the queue and all this would happen in real time Kafka allows for replaying events as it maintains a log of all the events that have passed to it this can be useful for debugging recovery during system failures or different use cases such as machine learning that may need access to its data in different ways how long you can keep your data can be configured using a retention policy integrating Kafka with your application can take it to the next level a distributed streaming platform like Apache Kafka can create a robust and resilient architecture for real-time data processing streaming can perform machine learning real-time anomaly detection and data enrichment data enrichment is simply taking data from one system and then adding data to it from another source suppose you have a database that stores user profiles and another that stores customer order data this data is combined into a single data structure and returned to a consumer such as the user's current cart the shopping cart service would subscribe to the user profile and transaction topics this data could then be enriched inside the service to display a profile and Order data a combination of decoupled systems coming together you can update Legacy systems with minimal Interruption to users transform and filter data in real time and connect systems through a publish And subscribe model a good thing to know about Apache Kafka is that it uses connectors to connect to external systems Source connectors can extract data from a database send it to Apache Kafka and then a sync connector can take that data from Apache Kafka and put it into another database syncs usually act on the data you are working with in some way you can track changes to the database using change data capture CDC which comes in two flavors either you can use CDC logs or CDC query CDC logs examine the database transaction or bin logs the types of logs will vary depending on the database logs are performant and aware of deletes in the database however it can require special permissions to set up you can also use CDC query it's less performant and relies on sending a query because of this it can only know what is inside of the database earlier I mentioned spring Cloud dataflow spring cloud dataflow is a powerful Cloud native toolkit that allows for developing and managing data integration and real-time data processing and pipelines if you're building a spring boot application it's a no-brainer organizations deal with massive volumes of data and the ability to efficiently process transform and analyze data is critical now spring Cloud dataflow provides a comprehensive solution to these challenges offering a flexible and scalable platform for building and managing data Pipelines it has security by standardizing and oau-2 and openid connect or authentication and authorization it enables continuous delivery and allows for upgrades and rollbacks of your Pipelines now data flow serves as a great way to enable an event-driven architecture it's also used with spring Cloud Stream app starters and has 60 pre-built sources so developers can kick-start their Solutions that's JMS Source mongodb rabbitmq gemfire cloud dataflow is also an open source framework built on top of the popular spring Boot and spring Cloud projects it provides a platform for data-driven microservice applications you can compose and orchestrate data processing pipelines known as streams and tasks based on applications to handle batch processing it supports multiple programming models such as Cloud stream and task the platform is also polygon supporting Kafka python.net and other technology polygot simply means that the platform can support multiple languages dataflow offers a plugable message broker including support for rapidmq Apache Kafka and Amazon kinesis it also has monitoring and management features to gain insights into the pipeline's performance monitor metrics troubleshoot issues and dynamically adjust the pipeline to optimize data processing it's a great tool for orchestrating streams you can have it batched in workflows scale data pipelines and orchestrate tasks combined with other tools and Technologies you can create an event-driven architecture with a rabbitmq or Apache Kafka and a spring cloud dataflow developing and learning on event driven architecture will always level up your skills I would be remiss if I didn't mention how you as a software engineer can get these Technologies in the cloud whether it's AWS Google cloud or Azure AWS offers Amazon mq Apache activemq rabbitmq Amazon Kinesis Google cloud has uh Cloud Pub sub Cloud dataflow Azure has Azure service bus event Hub and event grid as your skill set grows as software engineer you need to provide solutions to your company and the best solution may be to host your services and kubernetes or to move them to a managed service let me know what service you decided to go with and how you're implementing your event-driven architecture this has been another episode of the Joshua Matos show thanks for sticking around and I'll see you later
2023-07-28 21:51
