{"id":10672,"date":"2024-05-17T11:00:01","date_gmt":"2024-05-17T08:00:01","guid":{"rendered":"https:\/\/vasexperts.com\/?p=10672"},"modified":"2025-08-11T14:13:36","modified_gmt":"2025-08-11T11:13:36","slug":"processing-big-data-in-qoe-stor","status":"publish","type":"post","link":"https:\/\/vasexperts.com\/br\/blog\/dpi\/processing-big-data-in-qoe-stor\/","title":{"rendered":"Processamento de Big Data no QoE Stor"},"content":{"rendered":"O Big Data pode ser usado pelo operador nos seguintes cen\u00e1rios:\r\n<ol>\r\n \t<li><a href=\"https:\/\/vasexperts.com\/br\/blog\/quality-of-experience\/case-how-an-operator-can-reduce-the-uplink-cost\/\">Compreender a estrutura de tr\u00e1fego por protocolos e aplicativos e sua din\u00e2mica para formar planos tarif\u00e1rios atraentes, identificar pontos de peering e otimizar rotas.<\/a><\/li>\r\n \t<li><a href=\"https:\/\/wiki.vasexperts.com\/doku.php?id=dpi:qoe:use_cases:full_netflow_analytics:start#uplink_monitoring_service\">Monitorar a qualidade dos links de aplicativos para aplicativos espec\u00edficos e reagir rapidamente a problemas com o tr\u00e1fego da rede WAN.<\/a><\/li>\r\n \t<li><a href=\"https:\/\/vasexperts.com\/br\/blog\/dpi\/effective-network-monitoring-with-ssg-and-qoe-module-practical-case-study\/\">Identifique os assinantes problem\u00e1ticos com base na lat\u00eancia e nas solicita\u00e7\u00f5es de pacotes para solucionar problemas e melhorar a fidelidade do assinante.<\/a><\/li>\r\n \t<li><a href=\"https:\/\/wiki.vasexperts.com\/doku.php?id=dpi:qoe:use_cases:full_netflow_analytics:start#threats_monitor_service\">Monitore proativamente as amea\u00e7as cibern\u00e9ticas com base nas estat\u00edsticas de chamadas dos assinantes usando o banco de dados de feed da Kaspersky para reduzir o n\u00famero de BotNet na rede.<\/a><\/li>\r\n \t<li><a href=\"https:\/\/vasexperts.com\/br\/blog\/dpi\/what-ddos-attacks-are-and-how-a-telecom-operator-can-protect-against-them\/\">Monitore ataques DDoS e reaja a picos de tr\u00e1fego a tempo.<\/a><\/li>\r\n<\/ol>\r\n<h2>3 minutos para criar um relat\u00f3rio sobre 1 petabyte de dados<\/h2>\r\nVamos considerar o exemplo de uma operadora com 1 milh\u00e3o de assinantes. Em nossa experi\u00eancia, sup\u00f5e-se que uma operadora desse porte tenha cerca de 2 Tbps de tr\u00e1fego de pico.\r\n\r\nPara trabalhar com operadoras de casos comerciais t\u00edpicos:\r\n<ul>\r\n \t<li>Armazenar estat\u00edsticas IPFIX \u201cbrutas\u201d por 24 horas, o que equivale a aproximadamente    <strong>   45 TB.    <\/strong>  <\/li>\r\n \t<li>Em seguida, usando os algoritmos de QoE da Stor, os dados s\u00e3o agregados para armazenamento e reduzidos em um fator de 5. Normalmente, os dados agregados s\u00e3o armazenados por 3 meses, o que equivale a cerca de    <strong>   900 TB    <\/strong>   .<\/li>\r\n<\/ul>\r\nCombinado com outros tipos de dados (por exemplo, registro NAT ou registro GTP), nosso operador de refer\u00eancia chega a cerca de    <strong>   1 petabyte    <\/strong>   .\r\n\r\nEntretanto, n\u00e3o \u00e9 importante apenas obter os dados, mas tamb\u00e9m garantir que eles estejam dispon\u00edveis e sejam processados rapidamente. Os engenheiros e especialistas em marketing da operadora trabalham com eles todos os dias: eles criam relat\u00f3rios sobre v\u00e1rios campos e filtros para toda a profundidade do armazenamento de dados. Para o trabalho confort\u00e1vel dos usu\u00e1rios, o tempo de cria\u00e7\u00e3o de relat\u00f3rios por filtros n\u00e3o deve exceder    <strong>   3 minutos    <\/strong>   .\r\n\r\nAl\u00e9m disso, os dados do banco de dados s\u00e3o usados regularmente para enviar relat\u00f3rios peri\u00f3dicos aos servi\u00e7os da operadora por e-mail\/telegrama e para criar pain\u00e9is de controle.\r\n\r\nVoc\u00ea pode calcular o volume das estat\u00edsticas da sua operadora usando <a href=\"https:\/\/wiki.vasexperts.com\/doku.php?id=dpi:dpi_components:qoestor:install_and_update:hardware_recommendations:start\">a calculadora<\/a>.\r\n<h2>Componentes da solu\u00e7\u00e3o<\/h2>\r\nA aquisi\u00e7\u00e3o de dados ocorre em v\u00e1rios est\u00e1gios:\r\n<ol>\r\n \t<li>Passagem do tr\u00e1fego pelo SSG para analis\u00e1-lo por assinatura (protocolos e aplicativos).<\/li>\r\n \t<li>Envio de estat\u00edsticas do SSG por meio do protocolo IPFIX (NetFlow v10), que \u00e9 realizado por meio do balanceador ipfixcol2 para distribuir uniformemente as estat\u00edsticas entre os n\u00f3s e fornecer toler\u00e2ncia a falhas em caso de falha de um n\u00f3.<\/li>\r\n \t<li>Recebimento de estat\u00edsticas sobre o QoE Stor usando ipfixreceiver2.<\/li>\r\n<\/ol>\r\n<a href=\"\/wp-content\/uploads\/2024\/05\/com.png\" data-fancybox=\"\"><noscript><img loading=\"lazy\" decoding=\"async\" src=\"\/wp-content\/uploads\/2024\/05\/com-2-1.png\" alt=\"\" width=\"489\" height=\"586\" class=\"alignnone wp-image-10674\" srcset=\"\/wp-content\/uploads\/2024\/05\/com-2-1.png 816w, \/wp-content\/uploads\/2024\/05\/com-2-1-250x300.png 250w, \/wp-content\/uploads\/2024\/05\/com-2-1-768x920.png 768w\" sizes=\"(max-width: 489px) 100vw, 489px\"><\/noscript><img loading=\"lazy\" decoding=\"async\" src=\"\/wp-content\/uploads\/2024\/05\/com-2-1.png\" alt=\"\" width=\"489\" height=\"586\" class=\"alignnone wp-image-10674 lazyload\" sizes=\"(max-width: 489px) 100vw, 489px\" data-src=\"\/wp-content\/uploads\/2024\/05\/com-2-1.png\" data-srcset=\"\/wp-content\/uploads\/2024\/05\/com-2-1.png 816w, \/wp-content\/uploads\/2024\/05\/com-2-1-250x300.png 250w, \/wp-content\/uploads\/2024\/05\/com-2-1-768x920.png 768w\"><\/a>\r\n\r\nCaso o volume de tr\u00e1fego seja maior do que um \u00fanico SSG pode suportar, um cluster de DPI \u00e9 usado. O tr\u00e1fego \u00e9 extra\u00eddo da parte central da rede e enviado para o balanceador de carga do SSG, que distribui a carga uniformemente entre v\u00e1rios servidores SSG. O balanceador de carga \u00e9 capaz de lidar com at\u00e9 800 Gbps de tr\u00e1fego espelhado.\r\n\r\nInforma\u00e7\u00f5es mais detalhadas sobre a opera\u00e7\u00e3o do Load Balancer podem ser encontradas em nossa <a href=\"https:\/\/wiki.vasexperts.com\/doku.php?id=dpi:load_balancer:start\">base de conhecimento<\/a>.\r\n<h2><span style=\"font-weight: 400;\">Cluster de QoE<\/span><\/h2>\r\n<a href=\"\/wp-content\/uploads\/2024\/05\/qoe_cluster.png\" data-fancybox=\"\"><noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/qoe_cluster.png\" alt=\"\" width=\"356\" height=\"269\"><\/noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image- lazyload\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/qoe_cluster.png\" alt=\"\" width=\"356\" height=\"269\" data-src=\"\/wp-content\/uploads\/2024\/05\/qoe_cluster.png\"><\/a>\r\n\r\nO QoE Stor usa basicamente um banco de dados ClickHouse com a capacidade de criar um cluster de v\u00e1rios n\u00f3s:\r\n<ul>\r\n \t<li>Um n\u00f3 mestre \u00e9 atribu\u00eddo ao cluster, que recebe uma solicita\u00e7\u00e3o da GUI e envia solicita\u00e7\u00f5es ao n\u00f3 escravo.<\/li>\r\n \t<li>Cada n\u00f3 escravo cria um relat\u00f3rio com base em seus pr\u00f3prios dados e o envia para o n\u00f3 mestre.<\/li>\r\n \t<li>O n\u00f3 mestre agrega as respostas recebidas do n\u00f3 escravo e faz a representa\u00e7\u00e3o resultante para visualiza\u00e7\u00e3o na GUI.<\/li>\r\n<\/ul>\r\nEssa hierarquia permite realizar o dimensionamento linear do cluster quando novos n\u00f3s s\u00e3o adicionados, sem a necessidade de aumentar o desempenho do n\u00f3 mestre. A GUI trabalha com o cluster em um modo especial (ativado por uma op\u00e7\u00e3o separada nas configura\u00e7\u00f5es), modificando as consultas SQL para que os n\u00f3s criem relat\u00f3rios prontos para serem colados. Sem esse modo, o cluster \u00e9 apenas um armazenamento distribu\u00eddo, e o desempenho \u00e9 limitado apenas pelo desempenho do n\u00f3 mestre e pela largura de banda da rede entre os n\u00f3s de QoE.\r\n\r\n<a href=\"\/wp-content\/uploads\/2024\/05\/qoestor_arc.png\" data-fancybox=\"\"><noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/qoestor_arc.png\" alt=\"\" width=\"570\" height=\"750\"><\/noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image- lazyload\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/qoestor_arc.png\" alt=\"\" width=\"570\" height=\"750\" data-src=\"\/wp-content\/uploads\/2024\/05\/qoestor_arc.png\"><\/a>\r\n\r\nO QoE Stor executa o processamento de dados em v\u00e1rios est\u00e1gios:\r\n<ol>\r\n \t<li>ipfixreceiver2 recebe e grava dados brutos em um arquivo de texto com uma frequ\u00eancia especificada (10 seg.-10 min.) no disco padr\u00e3o.<\/li>\r\n \t<li>O p\u00f3s-processo realiza a agrega\u00e7\u00e3o de dados brutos para reduzir o espa\u00e7o de armazenamento e preencher tabelas para a cria\u00e7\u00e3o de relat\u00f3rios. A etapa de agrega\u00e7\u00e3o dura de 1 minuto a 1 hora.<\/li>\r\n<\/ol>\r\n<a href=\"\/wp-content\/uploads\/2024\/05\/storage.png\" data-fancybox=\"\"><noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/storage.png\" alt=\"\" width=\"311\" height=\"295\"><\/noscript><img loading=\"lazy\" decoding=\"async\" class=\" wp-image- lazyload\" title=\"\" src=\"\/wp-content\/uploads\/2024\/05\/storage.png\" alt=\"\" width=\"311\" height=\"295\" data-src=\"\/wp-content\/uploads\/2024\/05\/storage.png\"><\/a>\r\nV\u00e1rios tipos de discos s\u00e3o usados para otimizar o custo de armazenamento:\r\n<ul>\r\n \t<li>Padr\u00e3o \u2013 discos r\u00e1pidos para receber dados e executar o processo de agrega\u00e7\u00e3o; recomenda-se SSDs NVMe.<\/li>\r\n \t<li>hot \u2013 discos para armazenamento durante o per\u00edodo em que haver\u00e1 uma alta probabilidade de solicita\u00e7\u00e3o de relat\u00f3rios sobre esses dados, geralmente at\u00e9 3 meses.<\/li>\r\n \t<li>frio \u2013 discos lentos de grande volume para armazenamento de longo prazo, recomenda-se o uso de HDD.<\/li>\r\n<\/ul>\r\nO per\u00edodo de armazenamento em cada n\u00edvel \u00e9 definido na configura\u00e7\u00e3o via GUI. A movimenta\u00e7\u00e3o de dados entre discos e a limpeza de dados s\u00e3o feitas automaticamente de acordo com as configura\u00e7\u00f5es. H\u00e1 tamb\u00e9m um mecanismo de controle de estouro para proteger o banco de dados.\r\n\r\nPara obter mais informa\u00e7\u00f5es sobre as vantagens do Stingray Service Gateway e do m\u00f3dulo de an\u00e1lise de QoE, entre em contato com a VAS Experts. Deixe um <a href=\"https:\/\/vasexperts.com\/br\/test-us\/\">pedido de teste<\/a> para avaliar objetivamente os recursos e a funcionalidade do software.\r\n\r\n[subscription id=\u201d 11987\u2033]Assine o boletim informativo do blog para n\u00e3o perder novos materiais[\/subscription].","protected":false},"excerpt":{"rendered":"<p>Os clientes da VAS Experts s\u00e3o operadoras de telecomunica\u00e7\u00f5es que prestam servi\u00e7os aos assinantes finais, que, por sua vez, geram tr\u00e1fego.<br \/>\nPara gerenciar com efici\u00eancia esse fluxo de dados, a equipe da VAS Experts desenvolveu um sistema de monitoramento e an\u00e1lise de tr\u00e1fego, o Stingray Service Gateway (SSG), que permite \u00e0s operadoras n\u00e3o apenas analisar o tr\u00e1fego, mas tamb\u00e9m descarregar estat\u00edsticas.  \u00c0 medida que o n\u00famero de assinantes aumenta, as operadoras recebem mais e mais dados, o que complica o processo de armazenamento e processamento de dados.<\/p>\n<p> Neste artigo, analisaremos em detalhes como as informa\u00e7\u00f5es s\u00e3o processadas e armazenadas no servidor de estat\u00edsticas do SSG &#8211; QoE Stor.<\/p>\n","protected":false},"author":24,"featured_media":10680,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,44,55],"tags":[],"class_list":["post-10672","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dpi","category-quality-of-experience","category-functionality"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>(English) VASExperts<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/\",\"url\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/\",\"name\":\"[:br]Processamento de Big Data no QoE Stor[:fr]Traitement des donn\u00e9es massives dans le cadre du programme QoE Stor[:es]Procesamiento de macrodatos en QoE Stor[:en]Processing Big Data in QoE Stor[:] - (English) VASExperts\",\"isPartOf\":{\"@id\":\"https:\/\/vasexperts.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage\"},\"thumbnailUrl\":\"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg\",\"datePublished\":\"2024-05-17T08:00:01+00:00\",\"dateModified\":\"2025-08-11T11:13:36+00:00\",\"author\":{\"@id\":\"https:\/\/vasexperts.com\/#\/schema\/person\/f27674e15c2c6c09c36db4e68f9dd198\"},\"description\":\"In this article we will consider in detail how information is processed and stored in the SSG statistics server - QoE Stor.\",\"breadcrumb\":{\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#breadcrumb\"},\"inLanguage\":\"br-PT\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"br-PT\",\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage\",\"url\":\"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg\",\"contentUrl\":\"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg\",\"width\":1000,\"height\":460},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u0413\u043b\u0430\u0432\u043d\u0430\u044f \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0430\",\"item\":\"https:\/\/vasexperts.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Processing Big Data in QoE Stor\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/vasexperts.com\/#website\",\"url\":\"https:\/\/vasexperts.com\/\",\"name\":\"ITGLOBAL.COM\",\"description\":\"(English) VASExperts\",\"inLanguage\":\"br-PT\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/vasexperts.com\/#\/schema\/person\/f27674e15c2c6c09c36db4e68f9dd198\",\"name\":\"Darya Aulova\",\"url\":\"https:\/\/vasexperts.com\/br\/blog\/author\/darya-aulova\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"(English) VASExperts","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/","url":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/","name":"[:br]Processamento de Big Data no QoE Stor[:fr]Traitement des donn\u00e9es massives dans le cadre du programme QoE Stor[:es]Procesamiento de macrodatos en QoE Stor[:en]Processing Big Data in QoE Stor[:] - (English) VASExperts","isPartOf":{"@id":"https:\/\/vasexperts.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage"},"image":{"@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage"},"thumbnailUrl":"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg","datePublished":"2024-05-17T08:00:01+00:00","dateModified":"2025-08-11T11:13:36+00:00","author":{"@id":"https:\/\/vasexperts.com\/#\/schema\/person\/f27674e15c2c6c09c36db4e68f9dd198"},"description":"In this article we will consider in detail how information is processed and stored in the SSG statistics server - QoE Stor.","breadcrumb":{"@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#breadcrumb"},"inLanguage":"br-PT","potentialAction":[{"@type":"ReadAction","target":["https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/"]}]},{"@type":"ImageObject","inLanguage":"br-PT","@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#primaryimage","url":"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg","contentUrl":"\/wp-content\/uploads\/2024\/05\/frame-29787.jpg","width":1000,"height":460},{"@type":"BreadcrumbList","@id":"https:\/\/vasexperts.com\/blog\/quality-of-experience\/processing-big-data-in-qoe-stor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u0413\u043b\u0430\u0432\u043d\u0430\u044f \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0430","item":"https:\/\/vasexperts.com\/"},{"@type":"ListItem","position":2,"name":"Processing Big Data in QoE Stor"}]},{"@type":"WebSite","@id":"https:\/\/vasexperts.com\/#website","url":"https:\/\/vasexperts.com\/","name":"ITGLOBAL.COM","description":"(English) VASExperts","inLanguage":"br-PT"},{"@type":"Person","@id":"https:\/\/vasexperts.com\/#\/schema\/person\/f27674e15c2c6c09c36db4e68f9dd198","name":"Darya Aulova","url":"https:\/\/vasexperts.com\/br\/blog\/author\/darya-aulova\/"}]}},"_links":{"self":[{"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/posts\/10672"}],"collection":[{"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/comments?post=10672"}],"version-history":[{"count":12,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/posts\/10672\/revisions"}],"predecessor-version":[{"id":12784,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/posts\/10672\/revisions\/12784"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/media\/10680"}],"wp:attachment":[{"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/media?parent=10672"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/categories?post=10672"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vasexperts.com\/br\/wp-json\/wp\/v2\/tags?post=10672"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}